A method for evaluating the quality of formula raw materials based on multi-dimensional feature fusion

Abstract

The present invention discloses a method for evaluating the quality of formula raw materials by multi-dimensional feature fusion. First, the near-infrared spectrum and visual image data of the formula raw material sample to be tested are collected, and then the key spectrum bands are screened for the one-dimensional spectrum data, and then the visual two-dimensional spectrum image is located according to the screened key spectrum bands, and the pixels at the corresponding spectrum band positions are weighted enhanced. Finally, the one-dimensional and two-dimensional spectrum data and visual image data are respectively convoluted and feature fused through an automated search of a neural network structure, and the performance of the fused model is evaluated. The prediction model structure is determined through multiple iterations and the quality of the formula raw materials is predicted. Compared with the traditional single-dimensional data evaluation method, the multi-dimensional feature fusion method proposed in the present invention increases the spatial physical information of the formula raw material data, is more targeted, and integrates the molecular structure and visual characteristics of the formula raw materials, which is more in line with the complex scenarios of formula raw material evaluation, and ultimately improves the ability to predict the quality of the formula raw materials.

Description

Technical Field

[0001] The invention belongs to the technical field of formula industrial information and data processing, and in particular, relates to a formula raw material quality evaluation method based on multi-dimensional feature fusion. Background Art

[0002] With the development of sensor technology and the large-scale application of sensor products, the automation level of industrial data collection process has been continuously improved, and the formulation industry has accumulated a large amount of multi-source data resources. At the same time, the control of raw material quality in the formulation industry is particularly critical. The current methods for evaluating the quality of formulation raw materials include methods based on molecular spectroscopy, mass spectrometry and vision. Among them, computer vision and near-infrared spectroscopy analysis technology are very popular for their fast, low-cost, safe and non-destructive characteristics.

[0003] The current prediction methods are all single-dimensional modeling modes. Among them, the single-dimensional near-infrared spectrum ignores the spatial physical information of the spectrum. In addition, due to the long sequence of near-infrared spectra, the analysis of the full spectrum sequence requires repeated extraction of low-dimensional features and combination of high-dimensional features. At the same time, the spectral data contains not only key information but also some interference information and invalid information. The establishment of models with data containing interference information and invalid information will reduce efficiency and model prediction accuracy. The traditional modeling method based on the full spectrum segment cannot capture the spectrum segment with key features, and the modeling efficiency and accuracy are low.

[0004] As an important branch of artificial intelligence, computer vision has developed rapidly in recent years. The quality evaluation of formula raw materials based on vision has developed rapidly. For formula raw materials, single-dimensional visual features cannot fully capture the multi-level feature information of raw materials and have limitations. In addition, neural networks can quickly learn data distribution, fully mine the internal information of data, and perform deep feature extraction. However, the current modeling methods based on neural networks face the limitation of manual modeling. Summary of the invention

[0005] The present invention proposes a method for evaluating the quality of formula raw materials by multi-dimensional feature fusion, which selects the bands that are more critical to the current task target from the one-dimensional full-spectrum information of the spectrum, enhances the spectral features of the corresponding two-dimensional spectral image, extracts more critical and important information, and combines neural network structure search to achieve the modeling goal of multi-dimensional feature fusion for one-dimensional spectral data, two-dimensional spectral image data and two-dimensional visual image data, meets the needs of multi-dimensional feature fusion modeling of formula raw materials, and improves modeling efficiency and raw material quality prediction ability.

[0006] The present invention is implemented by the following technical solutions:

[0007] A multi-dimensional feature fusion method for formula raw material quality evaluation is proposed, including:

[0008] S1, collecting formula raw material data, including near infrared spectrum data and raw material visual image data;

[0009] S2, screening key spectral segments of one-dimensional spectral data;

[0010] S3, positioning the two-dimensional spectral image data based on the key spectrum bands screened out by S2, and performing image enhancement on the bands corresponding to the key spectrum bands;

[0011] S4, selecting a one-dimensional convolutional neural network for one-dimensional spectral data, and selecting a two-dimensional convolutional neural network for two-dimensional spectral image data and raw material visual image data, and performing neural network structure search respectively;

[0012] S5, using the one-dimensional convolutional neural network structure and the two-dimensional convolutional neural network structure obtained by searching in S4 to extract and fuse the multi-dimensional features of the formula raw materials;

[0013] S6, after feature fusion, obtain the prediction value according to different tasks and calculate the loss value, set the convergence condition, and determine the formula raw material prediction model after multiple iterations;

[0014] S7, according to the formula raw material prediction model obtained in S6, input the formula raw material data to be evaluated to obtain the quality evaluation prediction result.

[0015] In some embodiments of the present invention, step S1 includes:

[0016] Collect near infrared spectrum data and raw material visual image data of the samples to be tested;

[0017] Construct a one-dimensional spectral data set of near-infrared spectral samples;

[0018] The near-infrared spectral data is visualized and converted into images to establish a two-dimensional spectral image data set.

[0019] In some embodiments of the present invention, step S2 includes:

[0020] The Monte Carlo sampling method was used to sample the spectral samples and divide them into training set and test set. The PLS modeling method was used to build the model. Wavelength Variable The regression coefficient is , No. Wavelength Variable The weight of the regression coefficient is ;in, , Indicates the number of variables used in sampling;

[0021] The exponential decay function is used to eliminate the variables with small absolute value weight of regression coefficient; In the sampling modeling, the remaining variables after exponential decay eliminates the variables are , , ; is the sampling times;

[0022] Each time sampling is performed, according to the number of sampled variables Eliminate variables, establish a prediction model based on the selected variables, and calculate the cross-validation mean square error; Indicates the number of original wavelength variables;

[0023] In completion After sampling, select a set of spectral variable groups with the smallest cross-validation mean square error value. as the characteristic wavelength.

[0024] In some embodiments of the present invention, step S3 includes:

[0025] The key spectral segment set for one-dimensional spectral data modeling is obtained from step S2 ;gather The number of variables in is ,and Each spectral wavelength variable The weight is ;

[0026] According to the collection Medium wavelength variable Positioning in a 2D image , get the image coordinates The corresponding pixel value , and use the following formula to enhance the image pixels of the characteristic spectrum: ;

[0027] The enhanced pixel value Replace the original pixel value , get the enhanced image pixel value matrix .

[0028] In some embodiments of the present invention, step S5 includes:

[0029] The one-dimensional spectral data is subjected to a one-dimensional convolutional neural network operation to extract the feature matrix;

[0030] The enhanced two-dimensional spectral image data and the raw visual image data are subjected to a two-dimensional convolutional neural network operation to extract a feature matrix;

[0031] The one-dimensional convolution feature and the two-dimensional convolution feature data are fused to obtain a fused feature matrix.

[0032] In some embodiments of the present invention, in step S6, in calculating the loss value according to different tasks, the classification model is calculated as follows:

[0033] ;

[0034] The regression model is calculated as:

[0035] .

[0036] Compared with the prior art, the advantages and positive effects of the present invention are as follows: in the formula raw material quality evaluation method of multi-dimensional feature fusion proposed by the present invention, near-infrared spectrum and raw material visual image data of the raw material sample to be tested are collected, wherein the spectral data is converted into one-dimensional spectral data and visualized two-dimensional spectral image data, and then the one-dimensional spectral data is subjected to spectral segment screening with competitive adaptive weights, and is located in the visualized two-dimensional spectral image based on the key spectral segment, and the image features of the corresponding bands are enhanced; then a neural network structure search space is set, and one-dimensional and two-dimensional convolution operations are performed on the one-dimensional spectral data, the two-dimensional spectral image data and the raw material visual image data, respectively, and feature fusion is performed, and a multi-dimensional model for formula raw material quality prediction is established after multiple iterations. Based on the above, the present invention can perform feature fusion on one-dimensional spectral data, two-dimensional spectral image data and raw material visual image data, increase the spatial information of the spectrum and the visual information of the raw material, and provide richer feature information. Compared with the traditional one-dimensional model, the key spectral band image positioning and enhancement method in the present invention can better highlight the characteristics of the two-dimensional spectral image and enhance the key information. At the same time, the fusion of spectral and raw material visual image features makes the evaluation method of formula raw materials more three-dimensional, thereby improving the model feature extraction capability and prediction accuracy, and the automated neural network structure design breaks through the limitations of artificial neural network structure design.

[0037] Other features and advantages of the present invention will become more apparent after reading the detailed description of the embodiments of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0039] Figure 1 The figure is a schematic diagram of the processing steps of the method for evaluating the quality of formula raw materials by multi-dimensional feature fusion proposed by the present invention;

[0040] Figure 2The figure is a schematic diagram of the data processing flow of the multi-dimensional feature fusion formula raw material quality evaluation method proposed in the present invention. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] The present invention aims to propose a formula raw material quality evaluation method based on multi-dimensional feature fusion, make full use of multi-source data of formula raw materials, construct a formula raw material quality prediction model based on multi-dimensional feature fusion, and make full use of the neural network structure search method, ultimately solving the problems of incomplete single-dimensional characteristic information and limitations of artificial neural network design in the process of formula raw material quality evaluation.

[0043] Specifically, the multi-dimensional feature fusion formulation raw material quality evaluation method proposed in the present invention is as follows: Figure 1 and Figure 2 As shown, the following steps are included:

[0044] S1: Collect formula raw material data, including near-infrared spectrum data and raw material visual image data.

[0045] (1) Near-infrared spectral data and raw material visual image data are collected for the samples to be tested, and preprocessing operations are performed; at the same time, category labels are collected for classification tasks, and chemical composition or quality index data are collected for regression tasks; quality indicators include the chemical composition content in the cigarette formula industry, free amino nitrogen in the beer formula industry, etc.

[0046] (2) Constructing a one-dimensional spectral data set of near-infrared spectral samples of formula raw materials , and, construct a visual image data set of recipe ingredients ;in, represents the number of samples, Represents the near infrared spectrum data of a sample, represents the spectral data dimension, that is, the number of wavelength variables. Assume that the wavelength variable of the entire spectrum is , labels or quality indicators are expressed as . Then there is a one-dimensional data set The matrix is ​​as follows:

[0047] .

[0048] (3) Visualize the near-infrared spectral data, convert it into an image, and establish a two-dimensional spectral image data set ,in, , , Respectively represent the R / G / B component matrix of each pixel of the color spectrum image, with a value range of [0,255]. Assume that the pixel of the image is , then the pixel matrix of a sample is:

[0049] ,

[0050] ,

[0051] ,

[0052] but .

[0053] S2: Screening key spectral segments of one-dimensional spectral data.

[0054] (1) The Monte Carlo sampling method is used to sample spectral samples in a ratio of 8:2, where 80% of the samples are training sets and 20% are test sets. The PLS modeling method is used for modeling. is the sampling number, assuming that in each sampling process, the first Wavelength Variable The regression coefficient is , No. Wavelength Variable The weight of the regression coefficient is , then:

[0055]

[0056] in, , which indicates the number of variables used in sampling, that is, the number of near-infrared spectral points.

[0057] (2) Use the exponential decay function to eliminate the variables with relatively small absolute value weights of the regression coefficients, i.e., the spectral wavelength points; assuming that In the sampling modeling, the proportion of variables remaining after exponential decay eliminates variables is ,and

[0058]

[0059] in, and is a constant, calculated as follows: In the first sampling, all variables are used for modeling, so the proportion of retained variables is 1, After the sampling is completed, the number of variables retained is 2, and the ratio is ,therefore, and The calculation method is as follows:

[0060]

[0061]

[0062] (3) Each time sampling is performed, the number of sampling variables determined in step (2) is Variable elimination is performed, where It represents the number of original wavelength variables, establishes a prediction model based on the screening variables, and calculates RMSECV (mean square error of cross validation).

[0063] (4) After completion After sampling, select a set of spectral variable groups with the smallest RMSECV value as the characteristic wavelength.

[0064] S3: Position the two-dimensional spectral image data based on the key spectral bands screened out in S2, and perform image enhancement on the bands corresponding to the key spectral bands.

[0065] (1) Obtain the key spectral segment set for one-dimensional spectral data modeling from step S2 , assuming the set The number of variables in is ,and Each spectral wavelength variable The weight is .

[0066] (2) According to the collection in step (1) Medium wavelength variable Positioning in the two-dimensional spectral image , get the image coordinates The corresponding pixel value , and use formula (5) to enhance the image pixels of the characteristic spectrum, and use the enhanced pixel value Replace the original pixel value , get the enhanced image pixel value matrix .

[0067]

[0068] S4: Select a one-dimensional convolutional neural network for one-dimensional spectral data, and select a two-dimensional convolutional neural network for two-dimensional spectral image data and raw material visual image data, and perform neural network structure search respectively.

[0069] (1) Determine the neural network type.

[0070] The powerful feature extraction capability of convolutional neural networks is very suitable for feature extraction. One-dimensional convolutional neural networks can be effectively applied to one-dimensional sequence analysis, and can also be well used to analyze signal data with fixed-length cycles or fixed position expressions, such as near-infrared spectral data. However, for one-dimensional data analysis problems such as near-infrared spectroscopy, there is little research on the corresponding one-dimensional convolutional neural network structure. The applicability and effectiveness of convolutional neural networks for one-dimensional spectral data are difficult to determine. If the number of model layers is too shallow, it will not be able to learn enough training data and will cause underfitting problems. If the number of model layers is too deep, it will overfit and cause the generalization ability of the model to decrease. Therefore, exploring the appropriate network structure and number of layers is the key to modeling near-infrared spectra of raw materials.

[0071] In view of the above, the present invention selects a one-dimensional convolutional neural network operation for the one-dimensional spectral data representation, and selects a two-dimensional convolutional neural network operation for the two-dimensional image data (two-dimensional spectral image data and raw visual image data).

[0072] (2) Set the search space, controller parameters and search strategy.

[0073] For example: the search space can be defined as a DAG (directed acyclic graph) with Z nodes, and the controller is a cyclic network model whose parameters include the number of units and weights. The controller parameter update process involves generating sub-networks, training sub-networks, and evaluating sub-network performance (such as accuracy), and these performance indicators are fed back as signals to the controller model to optimize its parameters.

[0074] All network structures recommended by the controller can be regarded as subgraphs of the entire DAG, that is, they are all obtained by traversing the nodes on the DAG. At the same time, the activation function space can be set to Act={tanh, ReLU, sigmoid, Leaky ReLU, softmax,...}, and the calculation operation parameter set includes the number of convolution layers, convolution kernel size and pooling type, which can be set to Ope={3, 5, 7..., max pooling, average pooling}. Search strategies include reinforcement learning, evolutionary algorithms and gradient-based methods.

[0075] (3) Search and select the optimal one-dimensional convolutional neural network structure and two-dimensional convolutional neural network structure based on the set structure search parameters.

[0076] S5: Use the one-dimensional convolutional neural network structure and the two-dimensional convolutional neural network structure obtained by searching in S4 to extract and fuse the multi-dimensional features of the formula raw materials.

[0077] Specifically include:

[0078] (1) The one-dimensional spectral data is subjected to a one-dimensional convolutional neural network to complete feature extraction, and the enhanced two-dimensional spectral image data and raw visual image data are subjected to a two-dimensional convolutional neural network operation to complete feature extraction.

[0079] Among them, the feature matrix after one-dimensional spectral feature extraction is:

[0080] ;

[0081] in, Represents the feature dimension after one-dimensional convolution.

[0082] The feature matrix after two-dimensional image feature extraction is:

[0083] ;

[0084] in, Represents the feature dimension after two-dimensional convolution.

[0085] (2) Fuse the features after one-dimensional convolution with the feature data after two-dimensional convolution to obtain a fused feature matrix .

[0086] S6: After feature fusion, the prediction value is obtained according to different tasks and the loss value is calculated. The convergence conditions are set and the formula raw material prediction model is determined after multiple iterations.

[0087] After feature fusion, the activation function or fully connected layer is connected according to different tasks (classification task, regression task, etc.) to obtain the predicted value , and calculate the Loss value according to different tasks.

[0088] The following formula (6) is the calculation method for the classification model, and formula (7) is the calculation method for the regression model. Where n is the number of samples.

[0089]

[0090]

[0091] Set the convergence conditions and determine the formula raw material quality prediction model after multiple iterations.

[0092] S7: According to the formula raw material prediction model obtained in S6, the formula raw material data to be evaluated is input to obtain the quality evaluation prediction result.

[0093] The above-mentioned formula raw material quality evaluation method of multi-dimensional feature fusion proposed by the present invention, compared with the traditional single-dimensional modeling method, improves the multi-angle information understanding ability of the prediction model due to the integration of multi-source feature data, and increases the spatial physical feature information of the spectrum, and enhances the image features based on the one-dimensional key spectrum, so that the feature information is more prominent, and the feature extraction is more targeted. At the same time, compared with the traditional artificial neural network modeling method, the automated network structure search obtains the optimal network model, solves the limitation problem of manual modeling, and finally improves the modeling efficiency and model prediction performance. Therefore, the method proposed by the present invention is more suitable for formula raw material evaluation prediction model construction complex scenes.

[0094] The following is a comparison between the formula raw material quality evaluation method based on multi-dimensional feature fusion proposed in the present invention and the existing method.

[0095] As shown in Table 1, a one-dimensional spectral data model and a two-dimensional spectral image data model are compared with the method proposed in the present invention. First, taking the near-infrared spectral classification task as an example, using Accuracy, Precision, Recall, and F1-score as model performance measurement indicators, it can be seen that the performance of the method proposed in the present invention is better than that of the one-dimensional modeling method.

[0096] Table 1: Comparison between the one-dimensional data single model and the two-dimensional image single model and the method proposed in this invention

[0097]

[0098] Among them, IECSS-MFF is a formula raw material quality evaluation method based on multi-dimensional feature fusion proposed in the present invention.

[0099] Table 2 shows the comparison results between the multi-dimensional feature fusion method proposed in the present invention and the traditional modeling method. It can be seen that the classification effect of the method proposed in the present invention is better than that of the traditional modeling method.

[0100] Table 2: Comparison between the multi-dimensional feature fusion method proposed in this paper and the traditional modeling and classification method

[0101]

[0102] Among them, IECSS-MFF is a formula raw material quality evaluation method based on multi-dimensional feature fusion proposed in the present invention.

[0103] Table 3 compares the multidimensional feature fusion method proposed in the present invention with the traditional feature fusion method. It can be seen that the present invention plays a positive role in the feature fusion process. Compared with other conventional feature fusion methods, the features of each formula raw material level are more prominent and more targeted, and finally the prediction ability of the model is improved.

[0104] Table 3: Comparison between the multi-dimensional feature fusion method proposed in this paper and the traditional feature fusion method

[0105]

[0106] Among them, IECSS-MFF is a formula raw material quality evaluation method based on multi-dimensional feature fusion proposed in the present invention.

[0107] Table 4 compares the multidimensional feature fusion method proposed in the present invention with other methods in the task of predicting the chemical composition of formula raw materials. It can be seen that the method of the present invention is also applicable to processing regression tasks, and its prediction performance is better than other methods.

[0108] Table 4: Comparison of the proposed method with other methods in regression tasks

[0109]

[0110] Among them, IECSS-MFF is a formula raw material quality evaluation method based on multi-dimensional feature fusion proposed in the present invention.

[0111] It should be noted that, in the specific implementation process, the above-mentioned control part can be implemented by a hardware processor executing computer execution instructions in software form stored in the memory, which will not be elaborated here, and the programs corresponding to the actions performed by the above-mentioned control circuit can be stored in the computer-readable storage medium of the system in software form, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

[0112] The computer-readable storage medium mentioned above may include volatile memory, such as random access memory; may also include non-volatile memory, such as read-only memory, flash memory, hard disk or solid-state drive; may also include a combination of the above types of memory.

[0113] The processor mentioned above may also be a general term for multiple processing elements. For example, the processor may be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits, field programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc., or may be a dedicated processor.

[0114] It should be pointed out that the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by ordinary technicians in this technical field within the essential scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A method for evaluating the quality of formula raw materials by multi-dimensional feature fusion, characterized in that: include: S1, collecting formula raw material data, including near infrared spectrum data and raw material visual image data; S2, screening key spectral segments of one-dimensional spectral data; S3, positioning the two-dimensional spectral image data based on the key spectrum bands screened out by S2, and performing image enhancement on the bands corresponding to the key spectrum bands; S4, selecting a one-dimensional convolutional neural network for one-dimensional spectral data, and selecting a two-dimensional convolutional neural network for two-dimensional spectral image data and raw material visual image data, and performing neural network structure search respectively; S5, using the one-dimensional convolutional neural network structure and the two-dimensional convolutional neural network structure obtained by searching in S4 to extract and fuse the multi-dimensional features of the formula raw materials; S6, after feature fusion, obtain the prediction value according to different tasks and calculate the loss value, set the convergence condition, and determine the formula raw material prediction model after multiple iterations; S7, according to the formula raw material prediction model obtained in S6, input the formula raw material data to be evaluated to obtain the quality evaluation prediction result; Step S2 includes: The Monte Carlo sampling method was used to sample the spectral samples and divide them into training set and test set. The PLS modeling method was used to build the model. Wavelength Variable The regression coefficient is , No. Wavelength Variable The weight of the regression coefficient is ;in, , Indicates the number of variables used in sampling; The exponential decay function is used to eliminate the variables with small absolute value weight of regression coefficient; In the sampling modeling, the remaining variables after exponential decay eliminates the variables are , , ; is the sampling times; Each time sampling is performed, according to the number of sampled variables Eliminate variables, establish a prediction model based on the selected variables, and calculate the cross-validation mean square error; Indicates the number of original wavelength variables; In completion After sampling, select a set of spectral variable groups with the smallest cross-validation mean square error value. As characteristic wavelength; Step S3 includes: The key spectral segment set for one-dimensional spectral data modeling is obtained from step S2 ;gather The number of variables in is ,and Each spectral wavelength variable The weight is ; According to the collection Medium wavelength variable Positioning in a 2D image , get the image coordinates The corresponding pixel value , and use the following formula to enhance the image pixels of the characteristic spectrum: ; The enhanced pixel value Replace the original pixel value , get the enhanced image pixel value matrix ; Step S4 includes the steps of setting the search space, controller parameters and search strategy, including: the search space is defined as a directed acyclic graph with Z nodes, the controller is a cyclic network model, the parameters include the number of units and weights, the controller parameter update process involves generating sub-networks, training sub-networks and evaluating sub-network performance, and these performance indicators are fed back to the controller model as signals to optimize its parameters.

2. The method for evaluating the quality of formula raw materials by multi-dimensional feature fusion according to claim 1, characterized in that: Step S1 includes: Collect near infrared spectrum data and raw material visual image data of the samples to be tested; Construct a one-dimensional spectral data set of near-infrared spectral samples; The near-infrared spectral data is visualized and converted into images to establish a two-dimensional spectral image data set.

3. The method for evaluating the quality of formula raw materials by multi-dimensional feature fusion according to claim 1, characterized in that: Step S5 includes: The one-dimensional spectral data is subjected to a one-dimensional convolutional neural network operation to extract the feature matrix; The enhanced two-dimensional spectral image data and the raw visual image data are subjected to a two-dimensional convolutional neural network operation to extract a feature matrix; The one-dimensional convolution feature and the two-dimensional convolution feature data are fused to obtain a fused feature matrix.

4. The method for evaluating the quality of formula raw materials by multi-dimensional feature fusion according to claim 1, characterized in that: In step S6, the loss value is calculated according to different tasks, and the classification model is calculated as follows: ； The regression model is calculated as: 。

Images