A method and apparatus for determining the glutinousness of sorghum based on the coupling of spectral and visual data

Abstract

This invention relates to the field of raw grain analysis technology. In order to improve the accuracy of sorghum glutinousness discrimination, this application provides a method and device for sorghum glutinousness discrimination based on the coupling of spectral and visual data. By constructing a sorghum sample glutinousness database, a one-dimensional residual convolutional neural network is used to process mid-infrared spectral data. At the same time, a convolutional neural network is used to introduce the spatial structural features of visual images to achieve collaborative learning and feature complementarity of multimodal information. This not only overcomes the limitations of single-modal recognition of spectral or image data, but also improves the ability to extract complex features and the discrimination accuracy through deep learning models, enhances the stability and robustness of the model, and has stronger adaptability and generalization performance.

Description

Technical Field

[0001] This invention relates to the field of raw grain analysis technology, specifically a method and apparatus for determining the glutinousness of sorghum based on the coupling of spectral and visual data. Background Technology

[0002] Sorghum is an important raw material for brewing, food processing, and other fields. It can be divided into glutinous sorghum and non-glutinous sorghum, which differ significantly in starch composition, processing properties, and final product quality. For example, glutinous sorghum has a higher amylopectin content, making it suitable for specific brewing processes, while non-glutinous sorghum, with its higher amylose content, has different applications in food processing and feed production. Therefore, accurately distinguishing between glutinous and non-glutinous sorghum is crucial for brewing, the food industry, and agricultural cultivation.

[0003] CN117092061A discloses a method and system for rapidly determining the glutinous properties of sorghum. The method involves selecting several sorghum grains as standard samples to obtain their near-infrared spectral information, which is then used as the standard spectral information. Chemical analysis is used to determine the amylopectin and total starch content of the selected standard samples to calculate the glutinousness index, which is then used as the standard glutinousness index. A rapid detection model is established based on the standard spectral information and the standard glutinousness index. After obtaining the near-infrared spectral information of a batch of target samples, the established rapid detection model is used to determine the average glutinousness index of the corresponding batch of sorghum. The glutinous properties of the target samples are determined by comparing the average glutinousness index with a set discriminant value.

[0004] However, near-infrared spectroscopy reflects the combination and overtone information of molecular vibrations. While it has strong penetrating power, its chemical resolution is relatively low, limiting its ability to distinguish between samples with similar structures and subtle differences. In contrast, mid-infrared spectroscopy (MIR) provides higher-resolution molecular structure information, making it particularly suitable for distinguishing between glutinous sorghum and non-glutinous sorghum, which have subtle differences in chemical composition. However, relying solely on spectral data as a single modality presents challenges. Traditional modeling methods struggle to fully extract its deeper features, necessitating collaborative learning and feature complementarity based on multimodal information to improve the accuracy and robustness of identification. Summary of the Invention

[0005] To address the aforementioned issues, this application provides a method and apparatus for determining the glutinousness of sorghum based on the coupling of spectral and visual data.

[0006] The technical solution adopted by the present invention to solve the above problems is:

[0007] A method for determining the glutinousness of sorghum based on the coupling of spectral and visual data includes:

[0008] Construct a sample dataset, which includes mid-infrared spectral data and visual image data of glutinous sorghum and non-glutinous sorghum;

[0009] Based on a neural network model, a sorghum glutinousness discrimination model was created using mid-infrared spectral data and visual image data as input and sorghum glutinousness category as output; and the sorghum glutinousness discrimination model was trained using a sample dataset.

[0010] Acquire mid-infrared spectral data and visual image data of the sorghum samples to be evaluated; input them into the trained sorghum glutinousness discrimination model to obtain the sorghum glutinousness discrimination results.

[0011] Furthermore, the glutinousness discrimination model for sorghum includes a mid-infrared spectral data feature extraction module, a visual image data feature extraction module, and a multimodal feature coupling and classification decision module;

[0012] Mid-infrared spectral features were extracted from mid-infrared spectral data using a mid-infrared spectral data feature extraction module.

[0013] The visual image data feature extraction module is used to extract visual image data features from the visual image data.

[0014] The multimodal feature coupling and classification decision module concatenates features extracted from visual image data with features extracted from mid-infrared spectra, and generates the final category prediction result based on the MLP network.

[0015] Furthermore, mid-infrared spectral data were acquired using a Fourier transform infrared microscopy instrument; visual image data were acquired using a camera, maintaining the same spatial resolution, background color, and illumination conditions when acquiring visual image data.

[0016] Furthermore, before training the glutinousness discrimination model of sorghum, the following steps are also taken: preprocessing the sample dataset, specifically: denoising, baseline correction, smoothing and data standardization of the acquired mid-infrared spectral data; standardizing the three RGB channels of the visual image data; and data augmentation of the sample dataset.

[0017] Furthermore, data augmentation of the sample dataset specifically includes mid-infrared spectral data augmentation and / or visual image data augmentation.

[0018] Furthermore, mid-infrared spectral data augmentation was performed based on the mid-infrared spectral data of amylopectin and amylose. The specific steps are as follows:

[0019] Obtain mid-infrared spectral data of amylopectin and amylose;

[0020] One mid-infrared spectral data point of amylopectin and one mid-infrared spectral data point of amylose were randomly selected and augmented data were constructed using a weighted method. The weighting formula was: D = aX + bY, where D is one augmented data point, X is one mid-infrared spectral data point of amylopectin, Y is one mid-infrared spectral data point of amylose, and a and b are weighting coefficients, with a + b = 1. When constructing augmented data for glutinous sorghum, a was taken as a random value between 0.9 and 1.0; when constructing augmented data for non-glutinous sorghum, a was taken as a random value between 0.7 and 0.8.

[0021] The visual image data augmentation steps are as follows: randomly rotating, flipping, perspective transforming, adjusting brightness, adjusting contrast, adjusting saturation, and shifting tone of the visual image.

[0022] Furthermore, the specific process of training the glutinousness discrimination model of sorghum using the sample dataset is as follows: the sample dataset is divided into a training set and a test set. The training set is used to train the glutinousness discrimination model of sorghum, and the test set is used to test the training effect. When the error function of the test set is less than the error function threshold and the accuracy of the test set is higher than the accuracy threshold, the training is completed.

[0023] Furthermore, the error function is the cross-entropy loss function, expressed as: In the formula, M is the sample size; p ij Let be the true category label of the j-th sorghum category corresponding to the i-th sample; Let be the probability that the i-th sample is predicted to be the j-th sorghum category.

[0024] A sorghum waxiness discrimination device based on the coupling of spectral and visual data includes:

[0025] Dataset construction module: Constructs a sample dataset, which includes mid-infrared spectral data and visual image data of glutinous sorghum and non-glutinous sorghum;

[0026] Model building module: Based on a neural network model, a sorghum glutinousness discrimination model is created using mid-infrared spectral data and visual image data as input and sorghum glutinousness category as output; and the sorghum glutinousness discrimination model is trained using a sample dataset;

[0027] Category evaluation module: acquires mid-infrared spectral data and visual image data of the sorghum samples to be evaluated; inputs them into the trained sorghum glutinousness discrimination model to obtain the sorghum glutinousness discrimination results.

[0028] Furthermore, it also includes: a data preprocessing module: performing noise reduction, baseline correction, smoothing, and data standardization on the acquired mid-infrared spectral data; standardizing the RGB three channels of the visual image data; and performing data augmentation on the sample dataset.

[0029] The advantages of this invention compared to existing technologies are as follows: by constructing a glutinous database of sorghum samples, using a one-dimensional residual convolutional neural network to process mid-infrared spectral data, and simultaneously using a convolutional neural network to introduce spatial structural features of visual images, collaborative learning and feature complementarity of multimodal information are achieved; this not only overcomes the limitations of single-modal recognition of spectra or images, but also improves the ability to extract complex features and the accuracy of discrimination through deep learning models, enhances the stability and robustness of the model, and has stronger adaptability and generalization performance. Attached Figure Description

[0030] Figure 1 The flowchart shows the glutinousness discrimination method for sorghum based on the coupling of spectral and visual data.

[0031] Figure 2 This is a schematic diagram of the infrared spectrum of non-glutinous sorghum.

[0032] Figure 3 This is a schematic diagram of the infrared spectrum of glutinous sorghum.

[0033] Figure 4 This is a schematic diagram of the classification results for the test set without augmented data;

[0034] Figure 5 A schematic diagram of the classification results on the test set using augmented data;

[0035] Figure 6 This is a schematic diagram of a sorghum glutinousness discrimination device based on the coupling of spectral and visual data. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0037] like Figure 1 As shown, the glutinousness discrimination method for sorghum based on the coupling of spectral and visual data includes:

[0038] Construct a sample dataset, which includes mid-infrared spectral data and visual image data of glutinous sorghum and non-glutinous sorghum.

[0039] Mid-infrared spectral data of glutinous sorghum and non-glutinous sorghum were acquired using a Fourier transform infrared microscopy system, with a wavenumber range of 4000 cm⁻¹. -1 ~675cm -1 The mid-infrared spectral data includes wavenumber, wavelength, and corresponding transmittance. Visual image data of glutinous sorghum and non-glutinous sorghum were acquired using a camera. When acquiring visual image data, the same spatial resolution, the same background color, and the same lighting conditions must be maintained.

[0040] The mid-infrared spectra of non-glutinous sorghum and glutinous sorghum obtained in this embodiment are shown in the figure below. Figure 2 , 3 As shown, the two are quite similar, and it is difficult to distinguish them by the naked eye alone.

[0041] Therefore, the inventors created a sorghum glutinousness discrimination model based on a neural network model, using mid-infrared spectral data and visual image data as input and sorghum glutinousness category as output; and trained the sorghum glutinousness discrimination model using a sample dataset.

[0042] The glutinousness discrimination model for sorghum is a multimodal residual convolutional neural network coupled model, which includes a mid-infrared spectral data feature extraction module, a visual image data feature extraction module, and a multimodal feature coupling and classification decision module.

[0043] The mid-infrared spectral data feature extraction module is used to process mid-infrared spectral data. It takes mid-infrared spectral data as input and outputs mid-infrared spectral extracted features. The mid-infrared spectral data feature extraction module first constructs one-dimensional convolutional residual blocks based on one-dimensional convolution operations. Each residual block consists of a one-dimensional convolutional layer, a batch normalization layer, an activation function layer, a Dropout layer, and a skip connection method. The input data first undergoes an initial one-dimensional convolution operation to extract basic features, and then passes through a batch normalization layer and an activation function layer. Then, it passes through multiple one-dimensional convolutional residual blocks in sequence, and finally, a global adaptive average pooling method is used to achieve global feature aggregation.

[0044] The visual image data feature extraction module is used to process visual image data. It is built based on a two-dimensional convolutional neural network, which takes visual image data as input and outputs extracted features from the visual image data.

[0045] The multimodal feature coupling and classification decision module extracts features from visual image data, unfolds them into one-dimensional data, and concatenates them with features extracted from mid-infrared spectroscopy. The concatenated features are then fed into an MLP network, which generates the final category prediction result through a classification function.

[0046] This embodiment collected 811 sets of sample data, including 372 sets of non-glutinous sorghum data and 439 sets of glutinous sorghum data. Each set of data included mid-infrared spectral data and visual image data. 200 sets of data were randomly selected as the test set (100 sets of glutinous sorghum data and 100 sets of non-glutinous sorghum data), and the remaining 611 sets were used as the training set. The sorghum glutinousness discrimination model was trained using the training set, and the training effect was tested using the test set. Training was considered complete when the error function of the test set was less than the error function threshold and the accuracy of the test set was higher than the accuracy threshold. The error function adopted was the cross-entropy loss function, expressed as: In the formula, M is the sample size; p ijLet be the true category label of the j-th sorghum category corresponding to the i-th sample; Let be the probability that the i-th sample is predicted to be the j-th sorghum category. Other error functions can also be used; no restrictions are placed here. The classification results on the test set are as follows: Figure 4 As shown.

[0047] To improve the accuracy of the prediction results, this invention preprocesses the data in the sample dataset before model training, including: denoising, baseline correction, smoothing, and data standardization of the acquired mid-infrared spectral data; standardizing the RGB channels of the visual image data; and performing data augmentation on the sample dataset. Data augmentation includes mid-infrared spectral data augmentation and / or visual image data augmentation.

[0048] Since one of the main differences between glutinous sorghum and non-glutinous sorghum is the content of different types of starch, this embodiment uses mid-infrared spectral data augmentation based on amylopectin and amylose mid-infrared spectral data to improve the accuracy of sorghum category identification. The specific steps are as follows:

[0049] Mid-infrared spectral data of amylopectin and amylose were obtained using Fourier transform infrared microscopy. The wavenumber range of these spectra was consistent with that of glutinous sorghum and non-glutinous sorghum, also around 4000 cm⁻¹. -1 ~675cm -1 In this embodiment, 200 mid-infrared spectral data points of amylose and amylopectin were collected, with 100 data points each for amylose and amylopectin. One mid-infrared spectral data point of amylopectin and one mid-infrared spectral data point of amylose were randomly selected, and augmented data were constructed using a weighted method. The weighting formula is: D = aX + bY, where D is one augmented data point, X is one mid-infrared spectral data point of amylopectin, Y is one mid-infrared spectral data point of amylose, and a and b are weighting coefficients, with a + b = 1. The values ​​of a and b were determined based on the amylose and amylopectin content in glutinous sorghum and non-glutinous sorghum. In this embodiment, when constructing the augmented data for glutinous sorghum, a was a random value between 0.9 and 1.0; when constructing the augmented data for non-glutinous sorghum, a was a random value between 0.7 and 0.8. In this embodiment, the above data augmentation operation is performed randomly to obtain 5000 augmented data entries. During model training, n% of the data from the real dataset is randomly selected from the augmented data for data augmentation; in this embodiment, n% is 50%.

[0050] The visual image data augmentation steps are as follows: randomly rotating, flipping, perspective transforming, adjusting brightness, adjusting contrast, adjusting saturation, and shifting tone of the visual image.

[0051] The test set classification results after using augmented data are as follows: Figure 5As shown, it can be seen that after using data augmentation, 100% accuracy in classification can be achieved.

[0052] Acquire mid-infrared spectral data and visual image data of the sorghum samples to be evaluated; input them into the trained sorghum glutinousness discrimination model to obtain the sorghum glutinousness discrimination results.

[0053] Correspondingly, the present invention also provides a sorghum glutinousness discrimination device based on the coupling of spectral and visual data, such as... Figure 6 As shown, it includes:

[0054] Dataset construction module: Constructs a sample dataset, which includes mid-infrared spectral data and visual image data of glutinous sorghum and non-glutinous sorghum;

[0055] Model building module: Based on a neural network model, a sorghum glutinousness discrimination model is created using mid-infrared spectral data and visual image data as input and sorghum glutinousness category as output; and the sorghum glutinousness discrimination model is trained using a sample dataset;

[0056] Category evaluation module: acquires mid-infrared spectral data and visual image data of the sorghum samples to be evaluated; inputs them into the trained sorghum glutinousness discrimination model to obtain the sorghum glutinousness discrimination results.

[0057] Furthermore, it also includes: a data preprocessing module: performing noise reduction, baseline correction, smoothing, and data standardization on the acquired mid-infrared spectral data; standardizing the RGB three channels of the visual image data; and performing data augmentation on the sample dataset.

Claims

1. A method for determining the glutinousness of sorghum based on the coupling of spectral and visual data, characterized in that, include: Construct a sample dataset, which includes mid-infrared spectral data and visual image data of glutinous sorghum and non-glutinous sorghum; Based on a neural network model, a sorghum glutinousness discrimination model is created by taking mid-infrared spectral data and visual image data as input and sorghum glutinousness category as output. The glutinousness discrimination model of sorghum was trained using a sample dataset; Acquire mid-infrared spectral data and visual image data of the sorghum samples to be evaluated; Input it into the trained sorghum glutinousness discrimination model to obtain the sorghum glutinousness discrimination result; The sorghum glutinousness discrimination model includes a mid-infrared spectral data feature extraction module, a visual image data feature extraction module, and a multimodal feature coupling and classification decision module. Mid-infrared spectral features were extracted from mid-infrared spectral data using a mid-infrared spectral data feature extraction module. The visual image data feature extraction module is used to extract visual image data features from the visual image data. The multimodal feature coupling and classification decision module concatenates features extracted from visual image data with features extracted from mid-infrared spectra, and generates the final category prediction result based on an MLP network. Before training the sorghum glutinousness discrimination model, the following steps are also included: data augmentation of the sample dataset, including mid-infrared spectral data augmentation and / or visual image data augmentation; when performing mid-infrared spectral data augmentation based on amylopectin and amylose mid-infrared spectral data, the specific steps are as follows: Obtain mid-infrared spectral data of amylopectin and amylose; One mid-infrared spectral data point of amylopectin and one mid-infrared spectral data point of amylose were randomly selected, and augmented data were constructed according to a weighted method. The weighting formula is as follows: In the formula, D is an augmented data point, X is an infrared spectrum of amylopectin, Y is an infrared spectrum of amylose, a and b are weighting coefficients, and a+b=1; when constructing augmented data for glutinous sorghum, a takes a random value between 0.9 and 1.0; when constructing augmented data for non-glutinous sorghum, a takes a random value between 0.7 and 0.

8. The visual image data augmentation steps are as follows: randomly rotating, flipping, perspective transforming, adjusting brightness, adjusting contrast, adjusting saturation, and shifting tone of the visual image.

2. The sorghum glutinousness discrimination method based on the coupling of spectral and visual data according to claim 1, characterized in that, Mid-infrared spectral data were acquired using a Fourier transform infrared microscopy instrument; visual image data were acquired using a camera. When acquiring visual image data, the same spatial resolution, the same background color, and the same lighting conditions were maintained.

3. The sorghum glutinousness discrimination method based on the coupling of spectral and visual data according to claim 2, characterized in that, Before training the glutinousness discrimination model of sorghum, the following steps are also taken: preprocessing the sample dataset, specifically: denoising, baseline correction, smoothing and data standardization of the acquired mid-infrared spectral data; and standardizing the RGB three channels of the visual image data.

4. The sorghum glutinousness discrimination method based on the coupling of spectral and visual data according to claim 1, characterized in that, The specific process of training the glutinousness discrimination model of sorghum using the sample dataset is as follows: the sample dataset is divided into a training set and a test set. The training set is used to train the glutinousness discrimination model of sorghum, and the test set is used to test the training effect. The training is completed when the error function of the test set is less than the error function threshold and the accuracy of the test set is higher than the accuracy threshold.

5. The sorghum glutinousness discrimination method based on the coupling of spectral and visual data according to claim 4, characterized in that, The error function is the cross-entropy loss function, and its expression is: In the formula, M is the sample size; Let be the true category label of the j-th sorghum category corresponding to the i-th sample; Let be the probability that the i-th sample is predicted to be the j-th sorghum category.

6. A sorghum glutinousness discrimination device based on the coupling of spectral and visual data, used to implement the sorghum glutinousness discrimination method based on the coupling of spectral and visual data as described in any one of claims 1-5, characterized in that, include: Dataset construction module: Constructs a sample dataset, which includes mid-infrared spectral data and visual image data of glutinous sorghum and non-glutinous sorghum; Model building module: Based on a neural network model, using mid-infrared spectral data and visual image data as input and sorghum glutinousness category as output, a sorghum glutinousness discrimination model is created; The glutinousness discrimination model of sorghum was trained using a sample dataset; Category assessment module: Acquires mid-infrared spectral data and visual image data of the sorghum samples to be assessed; Input it into the trained sorghum glutinousness discrimination model to obtain the sorghum glutinousness discrimination result.

7. The sorghum glutinousness discrimination device based on the coupling of spectral and visual data according to claim 6, characterized in that, Also includes: Data preprocessing module: performs noise reduction, baseline correction, smoothing, and data standardization operations on the acquired mid-infrared spectral data; Standardize the RGB three channels of visual image data; Data augmentation is performed on the sample dataset.

Images