Spectral data intelligent processing method and system for food pesticide residue detection

Abstract

The invention discloses a spectral data intelligent processing method and system for food pesticide residue detection, and belongs to the technical field of spectral data processing. According to the spectral data intelligent processing method for food pesticide residue detection, by adopting an incremental tensor decomposition technology, dynamically increasing spectral data can be effectively processed, the timeliness and adaptability of data processing are improved, and the detection process is quicker and more sensitive. According to the method, a distributed computing framework is utilized, and through an optimized tensor division algorithm and a line computing strategy, the computation amount and communication overhead of an intermediate result are reduced, the efficiency of distributed tensor decomposition is improved, and efficient processing on a large-scale data set is realized. By combining an advanced spectral imaging technology and an intelligent data processing algorithm, the accuracy and reliability of pesticide residue detection are improved, and the dependence on professionals is reduced through an automatic and intelligent data processing flow, so that the detection process is more convenient and economical.

Description

Technical Field

[0001] The present invention belongs to the technical field of spectral data processing, and in particular relates to a spectral data intelligent processing method and system for food pesticide residue detection. Background Art

[0002] Spectral detection technology for food pesticide residues is an important technology in the field of food safety. With the development of spectral technology, especially hyperspectral imaging technology, it can provide rich spectral information to help us more accurately identify and quantify pesticide residues in food. Hyperspectral imaging technology collects image data of different wavelengths to form a hyperspectral cube, which enables us to analyze the chemical composition of the food surface at the molecular level and detect trace amounts of pesticide residues.

[0003] Although traditional spectral techniques have made some progress in pesticide residue detection, they still have limitations in processing speed, sensitivity and accuracy. For example, existing image data processing techniques require complex sample pre-processing steps and have high professional requirements for operators. In addition, as the amount of data continues to grow, the traditional full-volume tensor decomposition method is less efficient when processing dynamic data and cannot meet the needs of real-time or near-real-time analysis. Therefore, it is necessary to explore incremental tensor decomposition methods to adapt to real-time updates of data streams and dynamic adjustments of models. Summary of the invention

[0004] In view of the above situation, in order to overcome the defects of the prior art, the present invention adopts incremental tensor decomposition technology, which can effectively process dynamically growing spectral data, improve the timeliness and adaptability of data processing, and utilizes a distributed computing framework. Through the optimized tensor partitioning algorithm and row-based computing strategy, the computational complexity and communication overhead of intermediate results are reduced, and the efficiency of distributed tensor decomposition is improved.

[0005] In order to achieve the above object, the following technical scheme is adopted: The present invention provides a method for intelligent processing of spectral data for food pesticide residue detection, comprising the following steps:

[0006] Using hyperspectral imaging technology to obtain a spectral image of the surface of the food to be tested, forming an original spectral data set, wherein the image resolution meets a specific spectral resolution requirement;

[0007] Preprocessing of raw spectral datasets, including but not limited to spectral smoothing, baseline correction, normalization, and denoising, to improve data quality;

[0008] The PARAFAC decomposition model is used to reduce the noise of the preprocessed spectral data. The model formula is:

[0009]

[0010] Among them, X is the observed spectral matrix, which contains the spectral image data of the food surface, and A i is the score matrix of the i-th component, which contains the concentration information related to the component, B i is the loading matrix of the ith component, which contains the spectral feature information related to the component, ° represents the Hadamard product, R is the number of components, which represents the number of independent components assumed in the model, and E is the residual matrix, which represents the part of the spectral data that the model cannot explain;

[0011] The Isomap algorithm is used to maintain the nonlinear structure of the denoised spectral data in order to perform effective multilinear generalization processing. The Isomap algorithm maintains the local neighborhood relationship between data points to find the global low-dimensional embedding. The formula can be expressed as:

[0012]

[0013] Among them, d ij is the distance between the original data points, d′ ij It is the distance calculated by the Isomap algorithm, which takes into account the geometric structure of the data on the low-dimensional manifold;

[0014] When data is missing, incremental tensor decomposition is performed to update the tensor decomposition results without recalculating the entire data set. An online learning algorithm is used to adapt to the dynamic changes of data, which can be expressed as:

[0015] X new =A×B T +E,

[0016] Among them, X new is the newly added data block, A and B are the factor matrices that have been decomposed before, T is the core tensor, and E is the newly introduced residual;

[0017] Using the MLLFD algorithm, the basis vector contains the global and local features of the spectral curve, such as kurtosis and inflection points. The feature expression is enhanced by identifying the change pattern of the spectral curve. The basis vector can be expressed as:

[0018]

[0019] Among them, v i is the i-th basis vector, a ij and b ji are the decomposed scores and load vectors respectively;

[0020] Apply non-negative constraints to multilinear models to improve the interpretability and stability of the models, and use regularization techniques to reduce model overfitting;

[0021] The processed spectral data were used to conduct quantitative analysis of pesticide residues using quantitative analysis technology.

[0022] Furthermore, the pre-processing step further comprises:

[0023] Spectral smoothing: Savitzky-Golay filter is used to reduce random noise;

[0024] Baseline correction: polynomial fitting is used to eliminate spectral background interference;

[0025] Normalization: The minimum-maximum normalization method was used to normalize the spectral data;

[0026] Denoising: Wavelet transform method is used to further reduce the noise component in the data.

[0027] Furthermore, the iterative update process of the PARAFAC decomposition adopts an alternating least squares method, which specifically includes:

[0028] Initialize the score matrix and the loading matrix;

[0029] The score matrix and the load matrix are updated alternately until convergence or a predetermined number of iterations is reached;

[0030] Use the Frobenius norm as the loss function to minimize the reconstruction error;

[0031] The model complexity is controlled through regularization terms to prevent overfitting.

[0032] Furthermore, the multilinear generalization process includes:

[0033] Kernel MPCA, which maps data into a high-dimensional feature space by introducing kernel techniques to perform principal component analysis;

[0034] Manifold learning algorithms, such as LLE or Isomap, are used to preserve the topological structure of high-dimensional data in a low-dimensional space;

[0035] Multilinear algebraic decomposition technology is used to extract low-rank approximation of the data and retain the main information.

[0036] Furthermore, the incremental calculation of tensor decomposition includes:

[0037] Initial tensor decomposition, which decomposes the full dataset to obtain initial basis vectors and scores;

[0038] Incremental update: when new data arrives, only the new data is decomposed and fused with the old basis vectors;

[0039] Joint optimization of old and new data using recursive methods or gradient descent algorithms;

[0040] Balance computational efficiency and model accuracy by controlling update frequency and threshold.

[0041] Furthermore, the multilinear local eigendecomposition is achieved by the following steps:

[0042] Local feature extraction, identifying local change points in the spectral curve, such as peaks, valleys, and inflection points;

[0043] Local pattern analysis: clustering and pattern recognition of local change areas to extract local features;

[0044] Local feature fusion combines local features with global features to form a complete feature representation.

[0045] Furthermore, the class pattern and intra-class pattern basis vectors are constructed by the following steps:

[0046] Cluster analysis, which divides the spectral data into different categories, each representing a type of pesticide residue;

[0047] Pattern recognition, extracting features from the spectral data of each category and constructing class pattern basis vectors;

[0048] Local feature analysis,further analyzes the intra-class data and extracts the intra-class pattern basis,vectors to reflect the spectral changes under different conditions.

[0049] Furthermore, the multilinear model constraints are implemented by the following techniques:

[0050] Non-negativity constraints ensure that all scores and loads are non-negative, reflecting the actual physical meaning;

[0051] Tikhonov regularization controls model complexity and prevents overfitting.

[0052] On the other hand, the present invention also provides a spectral data intelligent processing system for food pesticide residue detection, comprising:

[0053] Hyperspectral imaging component, used to obtain spectral images of the surface of the food to be tested, with high resolution and high signal-to-noise ratio;

[0054] The preprocessing unit integrates multiple preprocessing algorithms and automatically selects the optimal preprocessing process;

[0055] The noise reduction unit uses the PARAFAC decomposition model for efficient noise reduction processing;

[0056] Multilinear generalization unit, integrating kernel MPCA and manifold learning algorithm to perform multilinear analysis of data;

[0057] Incremental tensor decomposition unit, designed to adapt to real-time updates of data streams to maintain the timeliness of the model;

[0058] Multilinear local feature decomposition unit, specially used to extract local features of spectral data;

[0059] Constraint application unit, integrating multiple regularization techniques to optimize model performance;

[0060] Quantitative analysis unit, integrating PCR and PLSR algorithms for precise quantitative analysis;

[0061] Data management and user interface, providing data storage, retrieval, analysis and user interaction functions.

[0062] Furthermore, the measurement system further comprises:

[0063] High-precision spectrometer with wide spectral range and high spectral resolution for accurate capture of spectral information;

[0064] The automated sample handling platform enables automatic sample collection, positioning and transfer, ensuring measurement consistency and repeatability.

[0065] The beneficial effects of the present invention are:

[0066] (1) The intelligent processing method of spectral data for food pesticide residue detection of the present invention can effectively process dynamically growing spectral data by adopting incremental tensor decomposition technology, thereby improving the timeliness and adaptability of data processing and making the detection process faster and more sensitive.

[0067] (2) The present invention utilizes a distributed computing framework and reduces the computational complexity and communication overhead of intermediate results through an optimized tensor partitioning algorithm and row-based computing strategy, thereby improving the efficiency of distributed tensor decomposition and achieving efficient processing on large-scale data sets.

[0068] (3) By combining advanced spectral imaging technology and intelligent data processing algorithms, the present invention not only improves the accuracy and reliability of pesticide residue detection, but also reduces the dependence on professionals through automated and intelligent data processing processes, making the detection process more convenient and economical. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 The present invention is a flow chart of a method for intelligently processing spectral data for detecting pesticide residues in food.

[0070] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention. DETAILED DESCRIPTION

[0071] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0072] Unless otherwise defined, all professional and scientific terms used herein have the same meanings as those familiar to those skilled in the art. In addition, any methods and materials similar or equivalent to those described herein may be applied to the present invention. The preferred implementation methods and materials described herein are for demonstration purposes only and are not intended to limit the content of this application.

[0073] The experimental methods in the following examples are conventional methods unless otherwise specified, and the experimental materials used in the following examples are purchased from commercial channels unless otherwise specified.

[0074] Example

[0075] A method and system for intelligent processing of spectral data for food pesticide residue detection.

[0076] The processing method comprises the following steps:

[0077] Using hyperspectral imaging technology to obtain a spectral image of the surface of the food to be tested, forming an original spectral data set, wherein the image resolution meets a specific spectral resolution requirement;

[0078] Preprocessing of raw spectral datasets, including but not limited to spectral smoothing, baseline correction, normalization, and denoising, to improve data quality;

[0079] The PARAFAC decomposition model is used to reduce the noise of the preprocessed spectral data. The model formula is:

[0080]

[0081] Among them, X is the observed spectral matrix, which contains the spectral image data of the food surface, and A i is the score matrix of the i-th component, which contains the concentration information related to the component, B i is the loading matrix of the ith component, which contains the spectral feature information related to the component, ° represents the Hadamard product, R is the number of components, which represents the number of independent components assumed in the model, and E is the residual matrix, which represents the part of the spectral data that the model cannot explain;

[0082] The Isomap algorithm is used to maintain the nonlinear structure of the denoised spectral data in order to perform effective multilinear generalization processing. The Isomap algorithm maintains the local neighborhood relationship between data points to find the global low-dimensional embedding. The formula can be expressed as:

[0083]

[0084] Among them, d ij is the distance between the original data points, d′ ij It is the distance calculated by the Isomap algorithm, which takes into account the geometric structure of the data on the low-dimensional manifold;

[0085] When data is missing, incremental tensor decomposition is performed to update the tensor decomposition results without recalculating the entire data set. An online learning algorithm is used to adapt to the dynamic changes of data, which can be expressed as:

[0086] X new =A×B T +E,

[0087] Among them, X new is the newly added data block, A and B are the factor matrices that have been decomposed before, T is the core tensor, and E is the newly introduced residual;

[0088] Using the MLLFD algorithm, the basis vector contains the global and local features of the spectral curve, such as kurtosis and inflection points. The feature expression is enhanced by identifying the change pattern of the spectral curve. The basis vector can be expressed as:

[0089]

[0090] Among them, v i is the i-th basis vector, a ij and b ji are the decomposed scores and load vectors respectively;

[0091] Apply non-negative constraints to multilinear models to improve the interpretability and stability of the models, and use regularization techniques to reduce model overfitting;

[0092] The processed spectral data were used to conduct quantitative analysis of pesticide residues using quantitative analysis technology.

[0093] The pre-processing step further comprises:

[0094] Spectral smoothing: Savitzky-Golay filter is used to reduce random noise;

[0095] Baseline correction: polynomial fitting is used to eliminate spectral background interference;

[0096] Normalization: The minimum-maximum normalization method was used to normalize the spectral data;

[0097] Denoising: Wavelet transform method is used to further reduce the noise component in the data.

[0098] The iterative update process of the PARAFAC decomposition adopts the alternating least squares method, which specifically includes:

[0099] Initialize the score matrix and the loading matrix;

[0100] The score matrix and the load matrix are updated alternately until convergence or a predetermined number of iterations is reached;

[0101] Use the Frobenius norm as the loss function to minimize the reconstruction error;

[0102] The model complexity is controlled through regularization terms to prevent overfitting.

[0103] The multilinear generalization process includes:

[0104] Kernel MPCA, which maps data into a high-dimensional feature space by introducing kernel techniques to perform principal component analysis;

[0105] Manifold learning algorithms, such as LLE or Isomap, are used to preserve the topological structure of high-dimensional data in a low-dimensional space;

[0106] Multilinear algebraic decomposition technology is used to extract low-rank approximation of the data and retain the main information.

[0107] The incremental calculation of tensor decomposition includes:

[0108] Initial tensor decomposition, which decomposes the full dataset to obtain initial basis vectors and scores;

[0109] Incremental update: when new data arrives, only the new data is decomposed and fused with the old basis vectors;

[0110] Joint optimization of old and new data using recursive methods or gradient descent algorithms;

[0111] Balance computational efficiency and model accuracy by controlling update frequency and threshold.

[0112] The multilinear local eigendecomposition is achieved by the following steps:

[0113] Local feature extraction, identifying local change points in the spectral curve, such as peaks, valleys, and inflection points;

[0114] Local pattern analysis: clustering and pattern recognition of local change areas to extract local features;

[0115] Local feature fusion combines local features with global features to form a complete feature representation.

[0116] The class pattern and within-class pattern basis vectors are constructed by the following steps:

[0117] Cluster analysis, which divides the spectral data into different categories, each representing a type of pesticide residue;

[0118] Pattern recognition, extracting features from the spectral data of each category and constructing class pattern basis vectors;

[0119] Local feature analysis,further analyzes the intra-class data and extracts the intra-class pattern basis,vectors to reflect the spectral changes under different conditions.

[0120] The multilinear model constraints are implemented by the following techniques:

[0121] Non-negativity constraints ensure that all scores and loads are non-negative, reflecting the actual physical meaning;

[0122] Tikhonov regularization controls model complexity and prevents overfitting.

[0123] On the other hand, this embodiment also provides a spectral data intelligent processing system for food pesticide residue detection, including:

[0124] Hyperspectral imaging component, used to obtain spectral images of the surface of the food to be tested, with high resolution and high signal-to-noise ratio;

[0125] The preprocessing unit integrates multiple preprocessing algorithms and automatically selects the optimal preprocessing process;

[0126] The noise reduction unit uses the PARAFAC decomposition model for efficient noise reduction processing;

[0127] Multilinear generalization unit, integrating kernel MPCA and manifold learning algorithm to perform multilinear analysis of data;

[0128] Incremental tensor decomposition unit, designed to adapt to real-time updates of data streams to maintain the timeliness of the model;

[0129] Multilinear local feature decomposition unit, specially used to extract local features of spectral data;

[0130] Constraint application unit, integrating multiple regularization techniques to optimize model performance;

[0131] Quantitative analysis unit, integrating PCR and PLSR algorithms for precise quantitative analysis;

[0132] Data management and user interface, providing data storage, retrieval, analysis and user interaction functions.

[0133] The measuring system further comprises:

[0134] High-precision spectrometer with wide spectral range and high spectral resolution for accurate capture of spectral information;

[0135] The automated sample handling platform enables automatic sample collection, positioning and transfer, ensuring measurement consistency and repeatability.

[0136] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

[0137] The present invention and its implementation methods are described above, which is not restrictive. The drawings are only one of the implementation methods of the present invention, and the actual application is not limited thereto. In short, if ordinary technicians in the field are inspired by it and design methods and embodiments similar to the technical solution without creativity without departing from the purpose of the invention, they should all fall within the protection scope of the present invention.

Claims

1. A method for intelligent processing of spectral data for food pesticide residue detection, characterized in that: The steps include: Using hyperspectral imaging technology to obtain a spectral image of the surface of the food to be tested, forming an original spectral data set, wherein the image resolution meets a specific spectral resolution requirement; Preprocessing of raw spectral datasets, including but not limited to spectral smoothing, baseline correction, normalization, and denoising, to improve data quality; The PARAFAC decomposition model is used to reduce the noise of the preprocessed spectral data. The model formula is: Among them, X is the observed spectral matrix, which contains the spectral image data of the food surface, and A i is the score matrix of the i-th component, which contains the concentration information related to the component, B i is the loading matrix of the i-th component, which contains the spectral feature information related to the component. represents the Hadamard product, R is the number of components, which represents the number of independent components assumed in the model, and E is the residual matrix, which represents the part of the spectral data that the model fails to explain; The Isomap algorithm is used to maintain the nonlinear structure of the denoised spectral data in order to perform effective multilinear generalization processing. The Isomap algorithm maintains the local neighborhood relationship between data points to find the global low-dimensional embedding. The formula can be expressed as: Among them, d ij is the distance between the original data points, d′ ij It is the distance calculated by the Isomap algorithm, which takes into account the geometric structure of the data on the low-dimensional manifold; When data is missing, incremental tensor decomposition is performed to update the tensor decomposition results without recalculating the entire data set. An online learning algorithm is used to adapt to the dynamic changes of data, which can be expressed as: X new =A×B T +E, Among them, X new is the newly added data block, A and B are the factor matrices that have been decomposed before, T is the core tensor, and E is the newly introduced residual; Using the MLLFD algorithm, the basis vector contains the global and local features of the spectral curve, such as kurtosis and inflection points. The feature expression is enhanced by identifying the change pattern of the spectral curve. The basis vector can be expressed as: Among them, v i is the i-th basis vector, a ij and b ji are the decomposed scores and load vectors respectively; Apply non-negative constraints to multilinear models to improve the interpretability and stability of the models, and use regularization techniques to reduce model overfitting; The processed spectral data were used to conduct quantitative analysis of pesticide residues using quantitative analysis technology.

2. The method for intelligent processing of spectral data for detecting pesticide residues in food according to claim 1, characterized in that: The pre-processing step further comprises: Spectral smoothing: Savitzky-Golay filter is used to reduce random noise; Baseline correction: polynomial fitting is used to eliminate spectral background interference; Normalization: The minimum-maximum normalization method was used to normalize the spectral data; Denoising: Wavelet transform method is used to further reduce the noise component in the data.

3. The method for intelligent processing of spectral data for detecting pesticide residues in food according to claim 2, characterized in that: The iterative update process of the PARAFAC decomposition adopts the alternating least squares method, which specifically includes: Initialize the score matrix and the loading matrix; The score matrix and the load matrix are updated alternately until convergence or a predetermined number of iterations is reached; Use the Frobenius norm as the loss function to minimize the reconstruction error; The model complexity is controlled through regularization terms to prevent overfitting.

4. The method for intelligent processing of spectral data for detecting pesticide residues in food according to claim 3, characterized in that: The multi-linear generalization process includes: Kernel MPCA, which maps data into a high-dimensional feature space by introducing kernel techniques to perform principal component analysis; Manifold learning algorithms, such as LLE or Isomap, are used to preserve the topological structure of high-dimensional data in a low-dimensional space; Multilinear algebraic decomposition technology is used to extract low-rank approximation of the data and retain the main information.

5. The method for intelligent processing of spectral data for detecting pesticide residues in food according to claim 4, characterized in that: The incremental calculation of tensor decomposition includes: Initial tensor decomposition, which decomposes the full dataset to obtain initial basis vectors and scores; Incremental update: when new data arrives, only the new data is decomposed and fused with the old basis vectors; Joint optimization of old and new data using recursive methods or gradient descent algorithms; Balance computational efficiency and model accuracy by controlling update frequency and threshold.

6. The method for intelligent processing of spectral data for detecting pesticide residues in food according to claim 5, characterized in that: The multilinear local eigendecomposition is achieved by the following steps: Local feature extraction, identifying local change points in the spectral curve, such as peaks, valleys, and inflection points; Local pattern analysis: clustering and pattern recognition of local change areas to extract local features; Local feature fusion combines local features with global features to form a complete feature representation.

7. The method for intelligently processing spectral data for detecting pesticide residues in food according to claim 6, characterized in that: The class pattern and within-class pattern basis vectors are constructed by the following steps: Cluster analysis, which divides the spectral data into different categories, each representing a type of pesticide residue; Pattern recognition, extracting features from the spectral data of each category and constructing class pattern basis vectors; Local feature analysis,further analyzes the intra-class data and extracts the intra-class pattern basis,vectors to reflect the spectral changes under different conditions.

8. The method for intelligently processing spectral data for detecting pesticide residues in food according to claim 7, characterized in that: The multilinear model constraints are implemented by the following techniques: Non-negativity constraints ensure that all scores and loads are non-negative, reflecting the actual physical meaning; Tikhonov regularization controls model complexity and prevents overfitting.

9. A spectral data intelligent processing system for food pesticide residue detection according to claim 8, characterized in that: include: Hyperspectral imaging component, used to obtain spectral images of the surface of the food to be tested, with high resolution and high signal-to-noise ratio; The preprocessing unit integrates multiple preprocessing algorithms and automatically selects the optimal preprocessing process; The noise reduction unit uses the PARAFAC decomposition model for efficient noise reduction processing; Multilinear generalization unit, integrating kernel MPCA and manifold learning algorithm to perform multilinear analysis of data; Incremental tensor decomposition unit, designed to adapt to real-time updates of data streams to maintain the timeliness of the model; Multilinear local feature decomposition unit, specially used to extract local features of spectral data; Constraint application unit, integrating multiple regularization techniques to optimize model performance; Quantitative analysis unit, integrating PCR and PLSR algorithms for precise quantitative analysis; Data management and user interface, providing data storage, retrieval, analysis and user interaction functions.

10. The spectral data intelligent processing system for food pesticide residue detection according to claim 9, characterized in that: The measuring system further comprises: High-precision spectrometer with wide spectral range and high spectral resolution for accurate capture of spectral information; The automated sample handling platform enables automatic sample collection, positioning and transmission, ensuring measurement consistency and repeatability.

Images