Spectral analysis noise reduction method and system based on structured spectral feature description

Through the generation and deduplication of structured spectral feature descriptors, the noise problem in near-infrared spectral analysis is solved, the accuracy and applicability of the analysis are improved, the stability of the model is enhanced, and the efficient noise reduction effect is achieved.

CN119027677APending Publication Date: 2024-11-26EAST CHINA UNIV OF SCI & TECH
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411208521.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-30
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

There are noise problems and inaccuracies of analysis results in near-infrared spectral analysis technology, resulting in low signal-to-noise ratio and poor model stability.

Method used

Using a noise reduction method based on structured spectral feature description, by obtaining near-infrared spectral data, extracting key points and spectral information, generating structured spectral feature descriptors, removing duplicate features, generating a reduced spectral matrix, and establishing a regression model To fit the physical and chemical properties of the sample.

Benefits of technology

It significantly improves the accuracy and applicability of near-infrared spectroscopy, reduces model complexity, enhances the stability and accuracy of the analysis process, and provides an efficient noise reduction solution.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027677A_ABST
    Figure CN119027677A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of near-infrared modeling data processing, in particular to a spectral analysis noise reduction method and system based on structured spectral feature description. The method comprises the following steps: acquiring near infrared spectrum data of a batch of samples, generating a near infrared spectrum image, and acquiring physicochemical property data of the samples; extracting key points of the near infrared spectrum image, extracting spectral information of the key points, adding the spectral information into corresponding feature descriptors, and generating structured spectral feature descriptors; according to the structured spectral feature descriptors, generating a spectral matrix after noise reduction; and establishing a regression model according to the denoised spectrum matrix and the physicochemical property data of the sample, inputting the denoised spectrum matrix of the sample to be analyzed to the regression model, and outputting a physicochemical property predicted value corresponding to the sample to be analyzed. Structured spectral feature description is adopted, effective noise reduction processing is achieved, and the performance of the model is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of near infrared modeling data processing, and more specifically, to a spectral analysis denoising method and system based on structured spectral feature description. Background Art

[0002] Near-infrared light is an electromagnetic wave between visible light and infrared light, and has important application value in the vibration of chemical bonds in organic molecules. This light band can detect the double frequency and combined frequency absorption of the stretching vibration of hydrogen groups (XH) in molecules. Near-infrared analysis technology is based on these absorption characteristics. Through chemometric methods and multivariate correction technology, a mathematical model between spectral information and sample properties is established to achieve accurate analysis of samples.

[0003] However, this analysis technique is often affected by factors such as spectral instrument noise and changes in the external environment in practical applications, resulting in low signal-to-noise ratios in some bands. This not only increases the difficulty of near-infrared spectral modeling, but also reduces the accuracy of the analysis. Therefore, it is crucial to denoise the spectrum. Reasonable denoising methods can effectively remove noise and retain valuable information, thereby simplifying the near-infrared regression model and enhancing its stability and reliability. In short, spectral denoising is a key technical means to improve the accuracy and applicability of near-infrared spectral analysis. Summary of the invention

[0004] A brief summary of one or more aspects is given below to provide a basic understanding of these aspects. This summary is not an exhaustive overview of all conceived aspects, and is neither intended to identify the key or critical elements of all aspects nor to define the scope of any or all aspects. Its only purpose is to give some concepts of one or more aspects in a simplified form as a prelude to a more detailed description that will be given later.

[0005] The purpose of the present invention is to provide a spectral analysis denoising method and system based on structured spectral feature description, so as to solve the noise problem existing in the existing near-infrared spectrum, the difficulty in establishing an accurate mathematical model and the inaccuracy of the analysis results.

[0006] The technical solution adopted by the present invention to solve the above technical problems is to provide a spectral analysis denoising method based on structured spectral feature description, comprising the following steps:

[0007] Step S1, obtaining near-infrared spectrum data of a batch of samples, generating near-infrared spectrum images, and obtaining physical and chemical property data of the samples;

[0008] Step S2, extracting key points of the near-infrared spectral image, and extracting spectral information of the key points, adding the spectral information to the corresponding feature descriptor, and generating a structured spectral feature descriptor;

[0009] Step S3, generating a denoised spectral matrix according to the structured spectral feature descriptor;

[0010] Step S4, establishing a regression model based on the spectral matrix after noise reduction and the physical and chemical property data of the sample to fit the relationship between the spectral matrix after noise reduction and the physical and chemical property data of the sample;

[0011] Step S5: input the spectral matrix of the sample to be analyzed after noise reduction into the regression model, and output the predicted value of the physical and chemical properties corresponding to the sample to be analyzed.

[0012] In one embodiment, in step S1, obtaining near infrared spectrum data of a batch of samples and generating near infrared spectrum images further comprises the following steps:

[0013] Step S11, preprocessing the near infrared spectrum data of the sample;

[0014] Step S12, generating a near infrared spectrum image;

[0015] Step S13, saving the near infrared spectrum image.

[0016] In one embodiment, it further comprises:

[0017] The near infrared spectrum data includes: wavelength and absorbance;

[0018] The preprocessing includes: deleting missing values ​​and smoothing noise;

[0019] The near infrared spectrum image is generated by programming, specifically including: reading near infrared spectrum data, extracting wavelength and absorbance data, and generating a near infrared spectrum graph;

[0020] The near infrared spectrum is stored in the form of a two-dimensional picture.

[0021] In one embodiment, the step S2 further includes the following steps:

[0022] Step S21, converting the near infrared spectrum image into a grayscale image, and calculating an integral image of the grayscale image;

[0023] Step S22: Based on the integral image, images of different scales are generated through Gaussian smoothing and downsampling operations, and a Gaussian pyramid is constructed to generate a multi-scale space;

[0024] Step S23, calculating the determinant of the Hessian matrix in different scale spaces, using the non-maximum suppression method to select key points that can maintain their feature strength and position consistency under various scale transformations;

[0025] Step S24, constructing a feature descriptor containing position, scale and direction information for each key point, determining a coverage area according to the position and scale, and extracting the spectral absorbance value of the pixel point from the coverage area;

[0026] Step S25: using a normalization processing method, generating a spectral information vector of the key point coverage area and adding it to the corresponding feature descriptor to generate a structured spectral feature descriptor containing spatial and spectral information.

[0027] In one embodiment, in step S2, extracting key points of the near-infrared spectral image further includes: extracting key points in the near-infrared spectral image using an accelerated robust feature method.

[0028] In one embodiment, the step S3 further includes the following steps:

[0029] Step S31, calculating similarity based on structured spectral feature descriptors and removing duplicate feature descriptors;

[0030] Step S32, extracting the spectral absorbance data of the key points from the deduplicated feature descriptors to determine the spectral band information;

[0031] Step S33: based on the spectral band information determined in step S32, the spectral absorbance data corresponding to the key points are deleted to generate a spectral matrix after noise reduction.

[0032] In one embodiment, the step S4 further includes: using multiple linear regression, partial least squares regression, support vector machine regression, and artificial neural network algorithm to establish a regression model and train the regression model.

[0033] In one embodiment, it further includes: using a KS algorithm based on Euclidean distance or a SPXY algorithm based on property variables to select m spectra with strong spectral representativeness from the sample set as a calibration set, and using the remaining samples as a validation set.

[0034] In one embodiment, the step S5 further includes: testing the near-infrared spectroscopy analysis prediction result through regression analysis test indicators, and the regression analysis test indicators include: prediction standard deviation and determination coefficient.

[0035] In order to achieve the above object, the present invention provides a spectral analysis denoising system based on structured spectral feature description, comprising:

[0036] a memory for storing instructions executable by a processor;

[0037] A processor is used to execute the instructions to implement any of the methods described above.

[0038] In order to achieve the above object, the present invention provides a computer-readable medium on which computer instructions are stored, wherein when the computer instructions are executed by a processor, any of the above methods is executed.

[0039] The spectral analysis denoising method and system based on structured spectral feature description proposed in the present invention adopts a structured spectral feature descriptor, which integrates the spatial features extracted by the accelerated robust feature algorithm and the spectral information of the key point coverage area. Through this method, the complexity of the modeling process can be effectively reduced, while the accuracy and applicability of near-infrared spectral analysis can be significantly improved, providing a more accurate and efficient denoising solution for near-infrared spectral analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The above features and advantages of the present invention can be better understood after reading the detailed description of the embodiments of the present disclosure in conjunction with the following drawings. In the drawings, the components are not necessarily drawn to scale, and components with similar related properties or features may have the same or similar reference numerals, wherein:

[0041] Figure 1 A step diagram of a spectral analysis denoising method based on structured spectral feature description according to an embodiment of the present invention is disclosed;

[0042] Figure 2 An original spectrum of a sample according to an embodiment of the present invention is disclosed;

[0043] Figure 3 A comparison chart of the determination coefficients of regression model test indicators before and after noise reduction according to an embodiment of the present invention is disclosed;

[0044] Figure 4 A comparison chart of predicted standard deviations of regression model test indicators before and after denoising according to an embodiment of the present invention is disclosed;

[0045] Figure 5 A principle block diagram of a spectral analysis denoising system based on structured spectral feature description according to an embodiment of the present invention is disclosed. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the invention and are not used to limit the invention.

[0047] Although the above methods are illustrated and described as a series of actions for simplicity of explanation, it should be understood and appreciated that these methods are not limited by the order of the actions, because according to one or more embodiments, some actions may occur in a different order and / or concurrently with other actions from those illustrated and described herein or not illustrated and described herein but understandable to those skilled in the art.

[0048] The spectral analysis denoising method and system based on structured spectral feature description proposed in the present invention not only extracts the spatial information of key points, but also integrates the spectral information of the areas covered by these key points by combining an accelerated robust feature algorithm. The generated structured spectral feature descriptor provides a multi-dimensional feature representation, which can reveal richer feature information in spectral image analysis.

[0049] In addition, the method provided by the present invention has multi-scale stability, which ensures that key points can be consistently detected at different scales. This is crucial for improving the accuracy of noise reduction processing, and also enhances the stability and accuracy of the entire spectral analysis process. Through this method, not only can the complexity of modeling be effectively reduced, but also the intelligent development of near-infrared spectroscopy analysis technology is provided.

[0050] The noise reduction method and system of the present invention not only improve the accuracy and applicability of near-infrared spectroscopy analysis through innovative structured feature description, but also open up a new path for technological progress and intelligent applications in this field.

[0051] Figure 1 The following is a step diagram of a spectral analysis denoising method based on structured spectral feature description according to an embodiment of the present invention. Figure 1 As shown, the spectral analysis denoising method based on structured spectral feature description proposed in the present invention comprises the following steps:

[0052] Step S1, obtaining near-infrared spectrum data of a batch of samples, generating near-infrared spectrum images, and obtaining physical and chemical property data of the samples;

[0053] Step S2, extracting key points of the near-infrared spectral image, and extracting spectral information of the key points, adding the spectral information to the corresponding feature descriptor, and generating a structured spectral feature descriptor;

[0054] Step S3, generating a denoised spectral matrix according to the structured spectral feature descriptor;

[0055] Step S4, establishing a regression model based on the spectral matrix after noise reduction and the physical and chemical property data of the sample to fit the relationship between the spectral matrix after noise reduction and the physical and chemical property data of the sample;

[0056] Step S5: input the spectral matrix of the sample to be analyzed after noise reduction into the regression model, and output the predicted value of the physical and chemical properties corresponding to the sample to be analyzed.

[0057] The present invention proposes a spectral analysis denoising method based on structured spectral feature description, aiming to establish an accurate and convenient regression model. First, an accelerated robust feature algorithm is applied to automatically identify key points in near-infrared spectral images to reduce reliance on manual experience. Subsequently, spectral information is extracted from these key points and integrated into feature descriptors to form structured spectral feature descriptors. Next, duplicate feature descriptors with the same spatial and spectral information are removed through deduplication processing. Finally, by deleting the absorbance data corresponding to the key points after deduplication, a denoised spectral matrix is ​​generated, which not only reduces noise interference, but also compresses the data dimension, effectively reducing the complexity of model training. The method of the present invention improves the accuracy and applicability of near-infrared spectral analysis, and provides an efficient and automated solution for the field of spectral analysis.

[0058] The above steps of the present invention will be described in detail below. It should be understood that within the scope of the present invention, the above technical features of the present invention and the technical features specifically described below (such as embodiments) can be combined with each other and interrelated to form a preferred technical solution.

[0059] Step S1, using a near infrared spectrometer to obtain near infrared spectrum data of a batch of samples, using the near infrared spectrum data to generate near infrared spectrum images, and obtaining the physical and chemical property data of these samples by chemical analysis methods. More specifically, the near infrared spectrum data of the samples are preprocessed, and then a near infrared spectrum graph is generated by a programming language and saved in the form of a two-dimensional image.

[0060] Among them, near-infrared spectral data include but are not limited to: wavelength and absorbance;

[0061] Preprocessing includes but is not limited to: removing missing values, smoothing noise, and ensuring that the data format is consistent and correct;

[0062] In this embodiment, the near infrared spectrum image can be generated by programming, including but not limited to: reading near infrared spectrum data, extracting wavelength and absorbance data, and generating near infrared spectrum graph;

[0063] In this embodiment, the near infrared spectrum diagram is saved in the form of a two-dimensional picture, but it is not limited to being saved in other forms.

[0064] Step S2: extract key points of each near-infrared spectral image, extract spectral information of the key points and add them to the corresponding feature descriptors to generate structured spectral feature descriptors.

[0065] The present invention adopts an accelerated robust feature method to extract key points in the near infrared spectrum image, wherein the accelerated robust feature is a robust image recognition and description algorithm.

[0066] Furthermore, step S2 includes the following steps:

[0067] Step S21: convert the input near-infrared spectrum image into a grayscale image, and calculate the integral image of the grayscale image to provide basic data for subsequent key point detection.

[0068] Step S22: Generate images of different scales on the integral image using Gaussian smoothing and downsampling techniques, and generate a multi-scale space by constructing a Gaussian pyramid.

[0069] Among them, Gaussian smoothing is an image processing technology that is suitable for eliminating Gaussian noise and is widely used in the noise reduction process of image processing; downsampling technology makes the image conform to the size of the display area and generates a thumbnail of the corresponding image.

[0070] Step S23: In multi-scale space, the determinant of the Hessian matrix is ​​calculated to evaluate the local stability of each position of the image. Through the non-maximum suppression method, key points whose feature strength and position remain consistent at different scales are selected.

[0071] Step S24: for each detected key point, construct a feature descriptor according to its position, scale and direction information, and determine a coverage area, and extract the spectral absorbance values ​​of all pixels in the area.

[0072] Step S25: normalize the spectral absorbance values ​​of the key point coverage area to generate a spectral information vector, and add the vector to the corresponding feature descriptor to form a structured spectral feature descriptor containing spatial and spectral information.

[0073] Among them, key points are detected by the determinant of the Hessian matrix at different scales, and the local gradient of the image is calculated at a scale of 20 × 20. By converting these gradients into feature descriptors, these descriptors can tolerate large local shape deformations and illumination changes, and the original feature descriptor dimension of each key point is 64 dimensions.

[0074] Step S3: Generate a denoised spectral matrix according to the structured spectral feature descriptor.

[0075] Furthermore, step S3 includes the following steps:

[0076] Step S31 : Evaluate the similarity of structured spectral feature descriptors extracted from a plurality of near-infrared spectral images by calculating the Euclidean distance between them.

[0077] The repeated feature descriptors whose spatial information and spectral information are completely consistent are removed to obtain a streamlined set of structured spectral feature descriptors.

[0078] Step S32: extract the spectral absorbance data of the key points from the deduplicated feature descriptors to determine the spectral band information and form a non-repetitive spectral band information set, laying the foundation for the next step of noise reduction processing.

[0079] Step S33: based on the spectral band information obtained in step S32, the spectral absorbance data corresponding to the key points are deleted to generate a spectral matrix after noise reduction.

[0080] After acquiring the near-infrared spectral data, it is first drawn into a spectral image, then key points are extracted based on the accelerated robust features, and structured spectral feature descriptors are generated. Finally, the denoised spectral matrix is ​​generated by deleting the absorbance data corresponding to the deduplicated key points. This process reduces the noise to a certain extent and compresses the dimension of the spectral matrix. Through steps S2 and S3, not only the interference of noise is reduced, but also the dimension of the data is compressed, reducing the complexity of subsequent model training.

[0081] Step S4: establishing a regression model based on the denoised spectral matrix and the physical and chemical property data of the sample.

[0082] The step S4 further includes but is not limited to: using multiple linear regression, partial least squares regression, support vector machine regression, and artificial neural network algorithm to establish a regression model and train the regression model.

[0083] The model input is the spectral matrix after noise reduction obtained in step S3, and the model output is the physical and chemical property data of the sample obtained in step S1.

[0084] In this embodiment, the regression model is a partial least squares regression model, which is established based on the spectral matrix after noise reduction and the physical and chemical property data of the sample.

[0085] Among them, partial least squares regression is a multivariate regression method based on principal component regression. While decomposing the spectral array and concentration array, it introduces the information of the concentration array into the spectral matrix decomposition process. It is the most commonly used modeling method in near-infrared modeling analysis.

[0086] Furthermore, the KS algorithm based on Euclidean distance or the SPXY algorithm based on property variables is used to select m spectra with strong spectral representativeness from the sample set as the calibration set, and the remaining samples are used as the validation set.

[0087] The principle of KS (Kennard-Stone) algorithm is to regard all samples as candidate samples of calibration set, and select samples from them into calibration set in turn. First, select the two samples with the farthest Euclidean distance into calibration set, calculate the Euclidean distance of each remaining sample to each known sample in calibration set, find the two samples farthest and closest to the selected samples, and select these two samples into calibration set, repeat the above steps until the number of samples reaches the requirement.

[0088] The SPXY (sample set partitioning based on joint xy distance) algorithm is developed based on the KS algorithm. The SPXY algorithm takes both the x variable and the y variable into account when calculating the distance between samples.

[0089] Step S5: Use a near-infrared spectrometer to collect near-infrared spectrum data of the sample to be analyzed, generate a spectrum image, use the above method to obtain a denoised spectrum matrix, input it into the regression model, and output the corresponding physical and chemical property prediction value.

[0090] Furthermore, the step S5 also includes: testing the near-infrared spectroscopy analysis prediction result by using regression analysis test indicators.

[0091] The regression analysis test indicators include but are not limited to: prediction standard deviation and determination coefficient.

[0092] The regression analysis test indicators include: prediction standard deviation RMSEP formula (1), determination coefficient R 2 Formula (2) and refer to the standard deviation of the data itself.

[0093] Forecast standard deviation

[0094] Where m is the number of spectra in the test set, y i,actual is the measured value of the ith spectral reference method, y i,predicted The predicted value of the i-th spectrum in the test set prediction process.

[0095] Coefficient of determination

[0096] Where n is the number of spectra in the test set, y i is the measured value of the ith spectral reference method, is the predicted value of the i-th spectrum in the test set prediction process, is the average of the values ​​determined by the reference method for all spectra in the test set.

[0097] Although the above methods are illustrated and described as a series of actions for simplicity of explanation, it should be understood and appreciated that these methods are not limited by the order of the actions, because according to one or more embodiments, some actions may occur in a different order and / or concurrently with other actions from those illustrated and described herein or not illustrated and described herein but understandable to those skilled in the art.

[0098] The spectral analysis denoising method based on structured spectral feature description proposed by the present invention is described below through an embodiment.

[0099] This embodiment uses a batch of crude oil as an experimental object to illustrate in detail the spectral analysis denoising method based on structured spectral feature description.

[0100] Step S1: Obtain 46 crude oil samples and record the corresponding physical and chemical properties. The specific process is as follows:

[0101] The spectral data of 46 samples were obtained by near-infrared spectrometer. The spectral data were expressed as wavelength and corresponding absorbance data. The spectral data were preprocessed to delete missing values ​​and smooth noise to ensure that the data format was consistent and correct.

[0102] The spectral data is read using programming language to extract wavelength and absorbance data. The number of data points in the spectral data string is 1037. These data strings are plotted into near-infrared spectral images, and the generated spectrum is saved as a two-dimensional image file.

[0103] like Figure 2 As shown, it is the original spectrum image of the sample. The horizontal axis represents the wave number (Wavelength), and the vertical axis represents the absorbance (Absorbance). The near-infrared spectrum image of the sample is matched with the density value in the laboratory analysis report.

[0104] Step S2: using the accelerated robust feature method to extract the key points of the near infrared spectrum image. The specific process is as follows:

[0105] Input 46 sample set image sample data, use accelerated robust feature method to extract the key points of near-infrared spectral images of 46 samples, and generate a (3112,64) matrix, where 3112 is the number of key points of all sample near-infrared spectral images.

[0106] The absorbance data of the spectral area covered by the key points is then extracted, normalized and added to the corresponding feature descriptor to form a higher-dimensional feature descriptor, and finally a structured spectral feature descriptor is generated.

[0107] Step S3: Generate a denoised spectral matrix based on the structured spectral feature descriptor. The specific process is as follows:

[0108] In this embodiment, the number of wavelength points of the sample spectrum is 1037. After the operation in the previous step, a feature descriptor containing spatial and spectral information is generated. For the structured spectral feature descriptors extracted from different images, the Euclidean distance is used to calculate the similarity, and the feature descriptors with consistent spectral information are removed to obtain a set of deduplicated structured spectral feature descriptors.

[0109] The spectral band information of the key points after deduplication is extracted, the absorbance data corresponding to the key points are deleted, and the denoised spectral matrix is ​​generated. The dimension of the new spectral matrix is ​​(46,533).

[0110] Step S4, establishing a regression model based on the spectral matrix after noise reduction and the physical and chemical properties of the sample;

[0111] The KS algorithm based on Euclidean distance was used. 30 representative samples were selected from the sample set as the calibration set, 7 samples as the validation set, and 9 samples as the test set. The calibration set samples were used to establish the model, the validation set samples were used to adjust the parameters, and the test set was used to test the accuracy of the model prediction.

[0112] The calibration set is spectrally preprocessed, and in this embodiment, a standard normal transformation is used. Then, a competitive adaptive reweighted sampling method is used to perform wavelength screening on the preprocessed spectrum, and then a partial least squares regression model is established based on the spectrum matrix after wavelength screening, and the physicochemical property (density) data is fitted. The generated partial least squares regression model is called the "model of this method".

[0113] In particular, as a control, the spectral matrix that has not been denoised by this method is preprocessed using the standard normal transformation spectrum, and then the wavelength of the preprocessed spectrum is screened using the competitive adaptive reweighted sampling method. A partial least squares regression model is then established based on the spectral matrix after wavelength screening, and the physicochemical properties (density) are fitted. The generated partial least squares regression model for control is called the "non-denoised model".

[0114] In order to demonstrate the applicability of the method, the wavelength screening was repeated 20 times using the competitive adaptive reweighted sampling method, including 10 times for the “present method model” and 10 times for the “no noise reduction model”.

[0115] Step S5: Import the 9 test samples of the test set into the "model of this method" and the "no noise reduction" model respectively, and predict the density attributes of the 9 samples respectively.

[0116] The evaluation results of the model are determined by the coefficient R 2 As shown in Table 1.

[0117] Table 1 Test set prediction accuracy R 2

[0118] serial number <![CDATA[This method model R 2 > <![CDATA[Un-denoised model R 2 > 1 0.79 0.70 2 0.76 0.59 3 0.79 0.63 4 0.79 0.77 5 0.77 0.68 6 0.75 0.59 7 0.78 0.65 8 0.79 0.72 9 0.82 0.70 10 0.87 0.66

[0119] like Figure 3 As shown, Figure 3 A comparison chart of the determination coefficient of the regression model test index before and after noise reduction according to an embodiment of the present invention is disclosed, where the horizontal axis represents the number of runs and the vertical axis represents the determination coefficient R of the evaluation result of the model. 2 , the solid line represents the R of the model of this method 2 The dashed line represents the R of the undenoised model. 2 As a result, the model of this method is superior to the model without denoising in both prediction effect and stability. As the number of runs increases, the prediction effect of the model of this method gradually improves, while the R 2 The value fluctuates greatly and the prediction effect is unstable.

[0120] The evaluation results of the model, prediction standard deviation RMSEP, are shown in Table 2.

[0121] Table 2 Test set prediction standard deviation RMSEP

[0122]

[0123] like Figure 4 As shown, Figure 4 The comparison chart of the predicted standard deviation of the regression model test index before and after denoising according to an embodiment of the present invention is disclosed. The horizontal axis represents the number of runs, the vertical axis represents the predicted standard deviation RMSEP of the model's evaluation results, the solid line represents the RMSEP value of the model of this method, and the dotted line represents the RMSEP value of the model without denoising. It can be seen that the model of this method is significantly better than the model without denoising in terms of prediction error. As the number of runs increases, the prediction accuracy of the model of this method gradually improves, while the RMSEP value of the model without denoising fluctuates greatly and the prediction error is large.

[0124] As shown in Tables 1 and 2 and Figure 3 and Figure 4 It can be seen that the model of this method improves the accuracy by 19% and reduces the prediction deviation by 20% compared with the non-denoising model.

[0125] Figure 5 The principle block diagram of a spectral analysis denoising system based on structured spectral feature description according to an embodiment of the present invention is disclosed. The spectral analysis denoising system based on structured spectral feature description may include an internal communication bus 501, a processor 502, a read-only memory (ROM)

[0126] 503, random access memory (RAM) 504, communication port 505, and hard disk 507. Internal communication bus 501 can realize data communication between components of spectral analysis noise reduction system based on structured spectral feature description. Processor 502 can make judgments and issue prompts. In some embodiments, processor 502 can be composed of one or more processors.

[0127] The communication port 505 can realize data transmission and communication between the spectral analysis and denoising system based on structured spectral feature description and external input / output devices. In some embodiments, the spectral analysis and denoising system based on structured spectral feature description can send and receive information and data from the network through the communication port 505. In some embodiments, the spectral analysis and denoising system based on structured spectral feature description can transmit and communicate data with external input / output devices in a wired form through the input / output terminal 506.

[0128] The spectral analysis denoising system based on structured spectral feature description may also include different forms of program storage units and data storage units, such as a hard disk 507, a read-only memory (ROM)

[0129] 503 and random access memory (RAM) 504, which can store various data files used for computer processing and / or communication, as well as possible program instructions executed by processor 502. Processor 502 executes these instructions to implement the main part of the method. The result processed by processor 502 is transmitted to an external output device through communication port 505 and displayed on the user interface of the output device.

[0130] For example, the implementation process file of the above-mentioned spectral analysis denoising method based on structured spectral feature description can be a computer program, which is stored in the hard disk 507 and can be recorded in the processor 502 for execution to implement the method of the present application.

[0131] When the implementation process file of the spectral analysis denoising method based on structured spectral feature description is a computer program, it can also be stored in a computer-readable storage medium as a product. For example, a computer-readable storage medium may include, but is not limited to, a magnetic storage device (e.g., a hard disk, a floppy disk, a magnetic strip), an optical disk (e.g., a compact disk (CD), a digital versatile disk (DVD)), a smart card, and a flash memory device (e.g., an electrically erasable programmable read-only memory (EPROM), a card, a stick, a key drive). In addition, the various storage media described herein can represent one or more devices and / or other machine-readable media for storing information. The term "machine-readable medium" may include, but is not limited to, wireless channels and various other media (and / or storage media) that can store, contain and / or carry code and / or instructions and / or data.

[0132] Compared with the prior art, the spectral analysis denoising method and system based on structured spectral feature description proposed in the present invention has the following effective effects:

[0133] 1) Compared with the traditional chemometrics application process of smoothing the data string and other noise reduction processes, the present invention uses the characteristic information of near-infrared spectral images to effectively filter out noise, thereby ensuring and improving the reliability and accuracy of the model.

[0134] 2) The key points extracted by the accelerated robust feature algorithm are combined with spectral information. The generated structured spectral feature descriptor contains both spatial information and spectral information, which improves the accuracy of noise recognition. The low dimension of feature information helps to reduce the complexity of the model, and has the advantages of strong adaptability and good fault tolerance, which can improve the prediction performance.

[0135] 3) This technology makes up for the lack of image analysis technology in near-infrared spectroscopy analysis to a certain extent, and confirms the feasibility of applying computer vision technology in the field of near-infrared spectroscopy. The exploration and development of this image analysis technology is a new opportunity to introduce new technologies into the field of near-infrared spectroscopy, which will help the development and application of near-infrared spectroscopy analysis technology in a broader, more accurate and convenient direction.

[0136] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0137] Those skilled in the art will appreciate that information, signals, and data may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips cited throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or optical particles, or any combination thereof.

[0138] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. The technician may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as resulting in a departure from the scope of the present invention.

[0139] The various illustrative logic modules and circuits described in conjunction with the embodiments disclosed herein may be implemented or executed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in cooperation with a DSP core, or any other such configuration.

[0140] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read and write information from / to the storage medium. In an alternative, a storage medium may be integrated into a processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and the storage medium may reside in a user terminal as discrete components.

[0141] The above embodiments are provided for persons familiar with the art to implement or use the present invention. Personnel familiar with the art can make various modifications or changes to the above embodiments without departing from the inventive concept of the present invention. Therefore, the protection scope of the present invention is not limited to the above embodiments, but should be the maximum scope of the innovative features mentioned in the claims.

Claims

1. A spectral analysis denoising method based on structured spectral feature description, characterized in that: The following steps are involved: Step S1, obtaining near-infrared spectrum data of a batch of samples, generating near-infrared spectrum images, and obtaining physical and chemical property data of the samples; Step S2, extracting key points of the near-infrared spectral image, and extracting spectral information of the key points, adding the spectral information to the corresponding feature descriptor, and generating a structured spectral feature descriptor; Step S3, generating a denoised spectral matrix according to the structured spectral feature descriptor; Step S4, establishing a regression model based on the spectral matrix after noise reduction and the physical and chemical property data of the sample to fit the relationship between the spectral matrix after noise reduction and the physical and chemical property data of the sample; Step S5: input the spectral matrix of the sample to be analyzed after noise reduction into the regression model, and output the predicted value of the physical and chemical properties corresponding to the sample to be analyzed.

2. The spectral analysis denoising method based on structured spectral feature description according to claim 1, characterized in that: In the step S1, the near infrared spectrum data of a batch of samples are obtained to generate a near infrared spectrum image, which further includes the following steps: Step S11, preprocessing the near infrared spectrum data of the sample; Step S12, generating a near infrared spectrum image; Step S13, saving the near infrared spectrum image.

3. The spectral analysis denoising method based on structured spectral feature description according to claim 2, characterized in that: Further including: The near infrared spectrum data includes: wavelength and absorbance; The preprocessing includes: deleting missing values ​​and smoothing noise; The near infrared spectrum image is generated by programming, specifically including: reading near infrared spectrum data, extracting wavelength and absorbance data, and generating near infrared spectrum graph; The near infrared spectrum is stored in the form of a two-dimensional picture.

4. The spectral analysis denoising method based on structured spectral feature description according to claim 1, characterized in that: In the step S2, extracting the key points of the near-infrared spectral image further includes: using an accelerated robust feature method to extract the key points in the near-infrared spectral image.

5. The spectral analysis denoising method based on structured spectral feature description according to claim 1, characterized in that: The step S2 further comprises the following steps: Step S21, converting the near infrared spectrum image into a grayscale image, and calculating an integral image of the grayscale image; Step S22: Based on the integral image, images of different scales are generated through Gaussian smoothing and downsampling operations, and a Gaussian pyramid is constructed to generate a multi-scale space; Step S23, calculating the determinant of the Hessian matrix in different scale spaces, using the non-maximum suppression method to select key points that can maintain their feature strength and position consistency under various scale transformations; Step S24, constructing a feature descriptor containing position, scale and direction information for each key point, determining a coverage area according to the position and scale, and extracting the spectral absorbance value of the pixel point from the coverage area; Step S25: using a normalization processing method, generating a spectral information vector of the key point coverage area and adding it to the corresponding feature descriptor to generate a structured spectral feature descriptor containing spatial and spectral information.

6. The spectral analysis denoising method based on structured spectral feature description according to claim 1, characterized in that: The step S3 further comprises the following steps: Step S31, calculating similarity based on structured spectral feature descriptors and removing duplicate feature descriptors; Step S32, extracting the spectral absorbance data of the key points from the deduplicated feature descriptors to determine the spectral band information; Step S33: based on the spectral band information determined in step S32, the spectral absorbance data corresponding to the key points are deleted to generate a spectral matrix after noise reduction.

7. The spectral analysis denoising method based on structured spectral feature description according to claim 1, characterized in that: The step S4 further includes: using multiple linear regression, partial least squares regression, support vector machine regression, and artificial neural network algorithm to establish a regression model and train the regression model.

8. The spectral analysis denoising method based on structured spectral feature description according to claim 7, characterized in that: The method further includes: using a KS algorithm based on Euclidean distance or a SPXY algorithm based on property variables to select m spectra with strong spectral representativeness from the sample set as a correction set, and using the remaining samples as a verification set.

9. The spectral analysis denoising method based on structured spectral feature description according to claim 1, characterized in that: The step S5 further includes: testing the near infrared spectrum analysis prediction result through regression analysis test indicators, and the regression analysis test indicators include: prediction standard deviation and determination coefficient.

10. A spectral analysis denoising system based on structured spectral feature description, comprising: a memory for storing instructions executable by a processor; A processor, configured to execute the instructions to implement the method according to any one of claims 1 to 9.

11. A computer-readable medium having computer instructions stored thereon, wherein when the computer instructions are executed by a processor, the method according to any one of claims 1 to 9 is executed.

Citation Information

Patent Citations

  • Method for constructing model for rapid and nondestructive real-time prediction of crude oil properties and application thereof

    CN107817223A

  • Near infrared spectrum quantitative analysis dimension reduction method and system based on random projection algorithm

    CN114676792A

  • Qualitative and quantitative analysis method and system based on near infrared spectrum image and medium

    CN116183547A

  • Near infrared spectrum analysis method and system based on image feature extraction

    CN117078960A