A method and system for identifying imperfect grains based on spectral space coupling features

Abstract

The present application belongs to the technical field of grain informatization processing, and particularly relates to a method and system for identifying imperfect grains based on spectral spatial coupling features. The method comprises inputting visible light images and hyperspectral data of the grains into a recognition model; a visible light image branch in the recognition model processes the visible light images of the grains to obtain the output of the branch; a hyperspectral data branch in the recognition model processes the hyperspectral data of the grains to obtain the output of the branch; the outputs of the visible light image branch and the hyperspectral data branch are used to obtain the recognition result; the recognition model is trained using grain sample images; the processing of the hyperspectral data branch comprises: expanding the hyperspectral data represented by a high-order tensor into a tensor factor multiplication mode containing spectral and spatial information features, and each mode is a tensor form in which all different spectral information features are superimposed; and the tensor form is subjected to feature extraction to obtain the branch output.

Description

Technical Field

[0001] This invention belongs to the field of grain information processing technology, specifically relating to an imperfect grain grain identification method and system based on spectral spatial coupling characteristics. Background Technology

[0002] With the continuous development of sensing technology, imaging technology, and intelligent vision algorithms, image recognition has demonstrated enormous application potential in the field of grain quality inspection. Especially in the identification of imperfect corn kernels, computer vision-based intelligent detection methods are gradually replacing traditional manual screening. By acquiring multimodal data of grain kernels through hyperspectral and visible light imaging systems, and combining preprocessing techniques such as filtering, denoising, and image enhancement, the clarity and feature discrimination of images can be significantly improved. Based on this, using deep learning models for high-dimensional feature extraction and intelligent recognition of kernel images enables automated identification and accurate judgment of imperfect kernels such as damaged, insect-eaten, and moldy kernels. This method not only effectively improves the efficiency and accuracy of detection but also enhances the robustness of the system under complex lighting conditions and sample differences.

[0003] Compared to traditional grain image acquisition systems, the introduction of intelligent sensing and intelligent detection technologies has enabled a higher degree of integration and automation in the detection process. Such systems typically use acquisition software to uniformly manage and control imaging equipment, achieving efficient acquisition and real-time storage of grain images. Based on the acquired data, the system can further perform tasks such as image processing, feature extraction, and quality assessment, thereby constructing an intelligent detection system that integrates acquisition, analysis, and detection.

[0004] During grain grain image acquisition, factors such as changes in ambient lighting, equipment disturbances, and calibration errors often lead to a decline in data quality, affecting the stability and automation level of the detection system. Traditional quality assessment methods mostly rely on manual experience or semi-automated detection, which suffers from low efficiency, high subjectivity, and insufficient accuracy, making it difficult to meet the needs of large-scale detection.

[0005] Chinese invention patent application CN120472149A discloses a rice pest and disease target detection algorithm based on Mamba and YOLOv8, including the following steps: acquiring RGB images of rice fields and preprocessing them to generate abnormal images; inputting the abnormal images into the MedVSSM module, extracting global long-distance dependency features through the Mamba state space model, and combining them with CNN to extract local features such as insect outlines and lesion textures, and fusing and outputting multi-scale features; performing cross-layer semantic enhancement and path aggregation through the C2F-MedVSSM module, and generating optimized feature maps through dynamic weight adjustment; outputting target detection boxes, categories, and confidence scores by the YOLOv8 module, and obtaining the final detection result after non-maximum suppression processing. This invention effectively improves the detection accuracy in scenarios with complex lighting and leaf occlusion by integrating the global modeling capabilities of Mamba and the local feature extraction advantages of CNN, and achieves feature complementarity through a dynamic weight mechanism, thus having practical application value for rice pest and disease control. Summary of the Invention

[0006] The purpose of this invention is to provide an imperfect grain identification method and system based on spectral spatial coupling characteristics, in order to solve the problem of insufficient accuracy of existing grain identification methods.

[0007] To achieve the above objectives, this invention provides an imperfect grain identification method based on spectral spatial coupling features, comprising:

[0008] The visible light image and hyperspectral data of the grain are input into the recognition model; the visible light image branch in the recognition model processes the visible light image of the grain to obtain the output of the branch;

[0009] The hyperspectral data branch in the recognition model processes the hyperspectral data of grains to obtain its output; the recognition result is obtained based on the outputs of the visible light image branch and the hyperspectral data branch; the recognition model is trained using grain sample images.

[0010] The processing method of the hyperspectral data branch includes: expanding the hyperspectral data represented by high-order tensors into tensor factors containing spectral and spatial information features of various modes, and multiplying them together, with each mode having a tensor form that superimposes all different spectral information features; and performing feature extraction on the tensor form to obtain the output of the branch.

[0011] The loss function during model training includes loss terms corresponding to the true labels of grain sample images and the labels predicted by the recognition model, and loss terms corresponding to the difference between the high-order tensor representation of the hyperspectral data and the tensor form.

[0012] Furthermore, the processing methods for visible light image branches include:

[0013] Different groups of dilated convolutional layers with varying dilation rates are used to process the normalized visible light image. The processing results of each group of dilated convolutional layers are weighted and summed, then processed by the FFN module, and finally fused with the processing results of the spatial state model branch to obtain the output of the visible light image branch. The processing of the spatial state model branch includes processing the normalized visible light image through the spatial state model, then fusing it with the unnormalized visible light image, and finally obtaining the processing result of the spatial state model branch based on the fusion result.

[0014] The weights when performing a weighted summation of the processing results from each dilated convolutional layer are adaptively adjusted during the training of the recognition model.

[0015] Furthermore, the expanded tensor form is subjected to a low-rank constraint in the gradient domain, wherein the product of the gradient operator along each mode and the tensor factor of that mode is equal to the product of the low-rank matrix of that mode and the Tikhonov matrix.

[0016] The loss function during model training also includes a loss term corresponding to the difference between the product of the gradient operator of each pattern and the tensor factor of that pattern and the product of the low-rank matrix of that pattern and the Tikhonov matrix.

[0017] Furthermore, the loss function during model training also includes a Tikhonov regularization loss term, which is obtained by summing the Tikhonov regularization results for each mode. The Tikhonov regularization results for each mode are obtained by weighted summing the results of taking the L2 norm of the low-rank matrix, the Tikhonov matrix, and the tensor factor respectively.

[0018] Furthermore, the methods for feature extraction from the tensor form include:

[0019] The tensor form is processed by 3D convolutional paths and 2D convolutional paths respectively, and the features output by the 3D and 2D convolutional paths are fused. Based on the feature fusion result, the output of the hyperspectral data branch is obtained.

[0020] The processing method of the 3D convolution path includes performing 3D convolution processing on the tensor form and superimposing the constant term of the 3D convolution path;

[0021] The processing method of the 2D convolution path includes performing 2D convolution processing on the dimensionality reconstruction result in the tensor form, and superimposing the constant term of the 2D convolution path.

[0022] Furthermore, the loss function during model training also includes a loss term corresponding to the difference between the expanded tensor form and the feature fusion result.

[0023] Furthermore, the methods for acquiring hyperspectral data of grains used as input to the recognition model include:

[0024] Background removal is performed on the acquired hyperspectral images of grain grains to obtain spectral data containing only the grain regions. Then, the image portion corresponding to each grain is separated from the whole image, and all pixel regions containing grains in the separated image portion are defined as regions of interest. Based on this, the average value of the pixel spectrum in the region of interest is calculated as the average spectral feature of the grain to obtain the hyperspectral data of grain grains used as input to the recognition model.

[0025] Furthermore, methods for acquiring hyperspectral images of grain grains include:

[0026] Each grain is randomly placed in a grid of a predetermined size and interval, facing the spectrometer in any orientation. The grid is then placed on a moving device illuminated by a light source. Once the moving device starts operating, the reflected light from the grain is acquired in real time by a spectral camera, generating a one-dimensional spatial image and corresponding spectral data. As the moving device moves continuously, the image sequence and spectral curve are continuously recorded. These are then stitched together to form a three-dimensional data cube containing a two-dimensional image and a spectral dimension, serving as the original hyperspectral image. The hyperspectral image of the grain is obtained based on the original hyperspectral image.

[0027] The processing methods for obtaining hyperspectral images of grains from the original hyperspectral images include standard black and white plate correction.

[0028] Furthermore, based on the feature fusion results, the output of the hyperspectral data branch can be obtained in the following ways:

[0029] The feature fusion result is subjected to depthwise separable convolution to obtain depthwise separable features; the result of the depthwise separable features after being processed by the channel attention mechanism is then fused with the depthwise separable features, and the output of the hyperspectral data branch is obtained based on the fusion result.

[0030] The technical solution described above provides a novel method for identifying imperfect grains based on spectral spatial coupling features. Its beneficial effects include: by dynamically adjusting the effective rank structure of the spectral tensor through tensor expansion, the feature representation capability is significantly enhanced, and more comprehensive inter-modal correlations can be captured. Adaptive fully connected tensor modeling can capture comprehensive modal correlations beyond the limitations of traditional adjacent modes, thereby significantly enhancing representation capability. Furthermore, by fully coordinating the tensor representation with the overall classification capability through losses during tensor expansion and losses in the classification results, the recognition accuracy of the identification model is improved.

[0031] This invention also provides an imperfect grain identification system based on spectral spatial coupling features, including a processor containing executable program instructions. These instructions are executed by the processor to implement the following imperfect grain identification method based on spectral spatial coupling features, specifically including:

[0032] The visible light image and hyperspectral data of the grain are input into the recognition model; the visible light image branch in the recognition model processes the visible light image of the grain to obtain the output of the branch;

[0033] The hyperspectral data branch in the recognition model processes the hyperspectral data of grains to obtain its output; the recognition result is obtained based on the outputs of the visible light image branch and the hyperspectral data branch; the recognition model is trained using grain sample images.

[0034] The processing method of the hyperspectral data branch includes: expanding the hyperspectral data represented by high-order tensors into tensor factors containing spectral and spatial information features of various modes, and multiplying them together, with each mode having a tensor form that superimposes all different spectral information features; and performing feature extraction on the tensor form to obtain the output of the branch.

[0035] The loss function during model training includes loss terms corresponding to the true labels of grain sample images and the labels predicted by the recognition model, and loss terms corresponding to the difference between the high-order tensor representation of the hyperspectral data and the tensor form.

[0036] Furthermore, the processing methods for visible light image branches include:

[0037] Different groups of dilated convolutional layers with varying dilation rates are used to process the normalized visible light image. The processing results of each group of dilated convolutional layers are weighted and summed, then processed by the FFN module, and finally fused with the processing results of the spatial state model branch to obtain the output of the visible light image branch. The processing of the spatial state model branch includes processing the normalized visible light image through the spatial state model, then fusing it with the unnormalized visible light image, and finally obtaining the processing result of the spatial state model branch based on the fusion result.

[0038] The weights when performing a weighted summation of the processing results from each dilated convolutional layer are adaptively adjusted during the training of the recognition model.

[0039] Furthermore, the expanded tensor form is subjected to a low-rank constraint in the gradient domain, wherein the product of the gradient operator along each mode and the tensor factor of that mode is equal to the product of the low-rank matrix of that mode and the Tikhonov matrix.

[0040] The loss function during model training also includes a loss term corresponding to the difference between the product of the gradient operator of each pattern and the tensor factor of that pattern and the product of the low-rank matrix of that pattern and the Tikhonov matrix.

[0041] Furthermore, the loss function during model training also includes a Tikhonov regularization loss term, which is obtained by summing the Tikhonov regularization results for each mode. The Tikhonov regularization results for each mode are obtained by weighted summing the results of taking the L2 norm of the low-rank matrix, the Tikhonov matrix, and the tensor factor respectively.

[0042] Furthermore, the methods for feature extraction from the tensor form include:

[0043] The tensor form is processed by 3D convolutional paths and 2D convolutional paths respectively, and the features output by the 3D and 2D convolutional paths are fused. Based on the feature fusion result, the output of the hyperspectral data branch is obtained.

[0044] The processing method of the 3D convolution path includes performing 3D convolution processing on the tensor form and superimposing the constant term of the 3D convolution path;

[0045] The processing method of the 2D convolution path includes performing 2D convolution processing on the dimensionality reconstruction result in the tensor form, and superimposing the constant term of the 2D convolution path.

[0046] Furthermore, the loss function during model training also includes a loss term corresponding to the difference between the expanded tensor form and the feature fusion result.

[0047] Furthermore, the methods for acquiring hyperspectral data of grains used as input to the recognition model include:

[0048] Background removal is performed on the acquired hyperspectral images of grain grains to obtain spectral data containing only the grain regions. Then, the image portion corresponding to each grain is separated from the whole image, and all pixel regions containing grains in the separated image portion are defined as regions of interest. Based on this, the average value of the pixel spectrum in the region of interest is calculated as the average spectral feature of the grain to obtain the hyperspectral data of grain grains used as input to the recognition model.

[0049] Furthermore, methods for acquiring hyperspectral images of grain grains include:

[0050] Each grain is randomly placed in a grid of a predetermined size and interval, facing the spectrometer in any orientation. The grid is then placed on a moving device illuminated by a light source. Once the moving device starts operating, the reflected light from the grain is acquired in real time by a spectral camera, generating a one-dimensional spatial image and corresponding spectral data. As the moving device moves continuously, the image sequence and spectral curve are continuously recorded. These are then stitched together to form a three-dimensional data cube containing a two-dimensional image and a spectral dimension, serving as the original hyperspectral image. The hyperspectral image of the grain is obtained based on the original hyperspectral image.

[0051] The processing methods for obtaining hyperspectral images of grains from the original hyperspectral images include standard black and white plate correction.

[0052] Furthermore, based on the feature fusion results, the output of the hyperspectral data branch can be obtained in the following ways:

[0053] The feature fusion result is subjected to depthwise separable convolution to obtain depthwise separable features; the result of the depthwise separable features after being processed by the channel attention mechanism is then fused with the depthwise separable features, and the output of the hyperspectral data branch is obtained based on the fusion result.

[0054] The technical solution of the imperfect grain identification system based on spectral spatial coupling features described above in this invention can achieve the same beneficial effects as the imperfect grain identification method based on spectral spatial coupling features described above. Attached Figure Description

[0055] Figure 1 This is an example diagram illustrating the processing principle of the hyperspectral data branch of the identification model in the implementation of the imperfect grain identification method based on spectral spatial coupling features of the present invention.

[0056] Figure 2 This is an example diagram illustrating the overall processing principle of the identification model in the implementation of the imperfect grain identification method based on spectral spatial coupling features of the present invention.

[0057] Figure 3 This is an example diagram illustrating the processing principle of the visible light image branch of the identification model in an embodiment of the imperfect grain identification method based on spectral spatial coupling features of the present invention.

[0058] Figure 4 This is an example diagram of the system architecture used for image acquisition in the implementation of the imperfect grain grain identification method based on spectral spatial coupling features of the present invention.

[0059] Figure 5This is an example of hyperspectral data of grains used as input to the recognition model, obtained by processing the imperfect grain recognition method based on spectral spatial coupling features in an embodiment of the present invention.

[0060] Figure 6 This is a flowchart illustrating the imperfect grain identification method in the implementation of the imperfect grain identification method based on spectral spatial coupling features of the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0062] Implementation of an Imperfect Grain Identification Method Based on Spectral Spatial Coupling Features

[0063] This embodiment presents a technical solution for identifying imperfect grain kernels based on spectral spatial coupling features. Starting from the cross-modal characteristics of abundant spectral information, strong high-dimensional correlation, and significant texture and complex spatial structure of visible light images, it achieves high-precision characterization and robust identification of defect features in imperfect grains. Specifically, the method includes:

[0064] The visible light image and hyperspectral data of the grain are input into the recognition model; the visible light image branch in the recognition model processes the visible light image of the grain to obtain the output of the branch;

[0065] The hyperspectral data branch in the recognition model processes the hyperspectral data of grains to obtain its output; the recognition result is obtained based on the outputs of the visible light image branch and the hyperspectral data branch; the recognition model is trained using grain sample images.

[0066] The processing method of the hyperspectral data branch includes: expanding the hyperspectral data represented by high-order tensors into tensor factors containing spectral and spatial information features of various modes, and multiplying them together, with each mode having a tensor form that superimposes all different spectral information features; and extracting features from the expanded tensor form to obtain the output of this branch.

[0067] The loss function during model training includes loss terms corresponding to the true labels of grain sample images and the labels predicted by the recognition model, and loss terms corresponding to the difference between the high-order tensor representation and the expanded tensor form of the hyperspectral data.

[0068] Therefore, by dynamically adjusting the effective rank structure of the spectral tensor through tensor expansion, the feature representation capability is significantly enhanced, and a more comprehensive intermodal correlation can be captured. Adaptive fully connected tensor modeling can capture comprehensive modal correlations beyond the limitations of traditional adjacent modes, thus significantly enhancing representation capability. On this basis, by fully coordinating the tensor representation and the overall classification capability through the loss in the tensor expansion process and the loss in the classification recognition result, the recognition accuracy of the recognition model is improved. Hyperspectral data is essentially a high-order tensor (e.g., space × space × spectrum). The spatial dimension contains local structure, texture, and morphological information; the spectral dimension contains material composition and spectral correlation information. There is a high degree of correlation and redundancy between different spectral bands. Expanding the high-order tensor into the multiplication of tensor factors on multiple modes is essentially decomposing the "strongly coupled high-dimensional representation" into a "low-dimensional factor representation of multi-subspace collaborative modeling." This low-dimensional factor representation naturally possesses the characteristics of controllable structure, reconfigurable information, and adjustable rank. Furthermore, since each mode of hyperspectral data unfolding superimposes all different spectral information features, the effective rank is not fixed, but adapts to the data distribution and optimization process. That is, it can achieve the effect of dynamically adjusting the effective rank structure of the spectral tensor during training.

[0069] In the expanded form, the factor matrix of each mode corresponds to the global linear mapping under that mode; each column of the factor matrix interacts with all spectral / spatial features, that is, any spectral feature can be associated with any spatial location through the tensor kernel; therefore, the above operation of expanding the hyperspectral data represented by higher-order tensors is functionally equivalent to the global receptive field of a fully connected layer.

[0070] Hyperspectral images inherently possess high-dimensional structures and complex spectral-spatial correlations, making it difficult for traditional tensor modeling methods to fully characterize their multimodal coupling features. To address this challenge, obtain more discriminative structural representations in the spatial domain, further explore fine-grained differences in the spectral domain, and overcome the limitations of traditional tensor decomposition methods in terms of fixed rank, limited representation, and noise sensitivity, a preferred embodiment adaptively transforms high-order tensors... The hyperspectral image patch (i.e., hyperspectral data represented by a higher-order tensor) is decomposed into interconnected tensor factors, and the correlations between complex modes are explicitly captured through an adaptive fully connected structure. The expanded tensor form is then obtained. The specific expression is as follows:

[0071] ; The reconstructed tensor features encapsulate the underlying spectral-spatial information. During model training, the recognition model adaptively learns tensor factors associated with each pattern. It actively captures the inherent structural relationships without relying on traditional fully connected tensor decomposition methods.

[0072] To more effectively model the local spatial continuity and spectral smoothness of hyperspectral images, this embodiment proactively introduces an innovative gradient domain low-rank constraint on the tensor factor of the mode expansion. Specifically, the gradient domain low-rank constraint is applied to the expanded tensor form, and the constraint condition is that the product of the gradient operator along each mode and the tensor factor of that mode is equal to the product of the low-rank matrix of that mode and the Tikhonov matrix.

[0073] The loss function during model training also includes a loss term corresponding to the difference between the product of the gradient operator and the tensor factor of each pattern and the product of the low-rank matrix and the Tikhonov matrix of that pattern. Specifically, for the expanded matrix... The low-rank constraint in the gradient domain can be expressed as:

[0074]

[0075] in, It is the gradient operator along the k-th mode, used to capture local spatial continuity and spectral correlation. Low-rank matrix. With Tikhonov matrix It is used to actively reduce redundant information while enhancing the robustness of feature extraction.

[0076] To further stabilize the adaptive tensor decomposition and prevent overfitting when labeled data is limited, this embodiment also introduces Tikhonov regularization during the training of the recognition model. Specifically, the loss function during the training of the recognition model also includes a Tikhonov regularization loss term, which is obtained by summing the Tikhonov regularization results corresponding to each mode. The Tikhonov regularization results corresponding to each mode are obtained by weighted summation of the results of taking the L2 norm of the low-rank matrix, the Tikhonov matrix, and the tensor factor, respectively.

[0077] In a preferred embodiment, the Frobenius norm is specifically used, and the Tikhonov regularization result for each mode is... Specifically as follows:

[0078] ; in Let Frobenius norm be denoted, and N be the total number of patterns. All of these are regularization factors. This regularization actively controls the complexity of the recognition model, improves numerical stability, and significantly enhances the recognition model's ability to generalize to unseen data.

[0079] Furthermore, referring to Figure 1 In this embodiment, the method for feature extraction of the above-mentioned tensor form includes:

[0080] The expanded tensor form is processed by 3D convolution path and 2D convolution path respectively, and the features output by 3D and 2D convolution path are fused. Based on the feature fusion result, the output of the hyperspectral data branch is obtained.

[0081] The processing method for 3D convolution paths includes performing 3D convolution on the expanded tensor form mentioned above, and superimposing the constant term of the 3D convolution path.

[0082] The processing method for 2D convolution paths includes performing 2D convolution on the dimensionality reconstruction result of the expanded tensor form mentioned above, and superimposing the constant term of the 2D convolution path.

[0083] Based on this, refer to Figure 1 To address the limited computing resources in actual grain quality testing, this embodiment adopts a lightweight design strategy. Therefore, based on the feature fusion results, the output of the hyperspectral data branch is obtained in the following ways:

[0084] The feature fusion result is subjected to a depthwise separable convolution to obtain depthwise separable features. The result of processing the depthwise separable features through a channel attention mechanism is then fused with the depthwise separable features, and the output of the hyperspectral data branch is obtained based on the fusion result. These optimizations actively reduce computational complexity and memory consumption, enabling the tensor regularized network to efficiently adapt to the real-time inference requirements of edge devices.

[0085] In a preferred embodiment, the tensor regularization network of the hyperspectral data branch is designed as an end-to-end jointly optimized architecture, enabling heterogeneous training of tensor decomposition and feature extraction of visible light images.

[0086] Specifically, the tensor regularization network employs a dual convolutional path to directly utilize the unfolded tensor features for efficient feature integration. The 3D convolutional path directly processes the provided tensor features to preserve the complete spectral-spatial structural information. This is expressed as follows:

[0087]

[0088] 2D convolutional paths primarily process the spatial dimension of tensor features, actively extracting local spatial patterns and structural information, as specifically expressed below:

[0089]

[0090] In tensor regularized networks (i.e.) Figure 2The STM Model in the code fuses the outputs of the 3D and 2D convolutional paths, as specifically expressed below:

[0091]

[0092] In the depthwise separable convolution path, which includes depthwise separable convolution and channel attention mechanisms, standard convolution is decomposed into depthwise separable convolution. and channel attention mechanism In a two-step operation, the hyperspectral data branch network of the recognition model significantly reduces the number of parameters and computational cost while maintaining feature extraction capabilities. Such depthwise separable convolutional paths can be expressed as follows:

[0093] ; ;

[0094] but This is the output of the STM Model. For details on how to derive the hyperspectral data branch output from this output, please refer to [link / reference needed]. Figure 2 .

[0095] In this embodiment, the loss function during model training also includes a loss term corresponding to the difference between the expanded tensor form and the feature fusion result. In summary, during the heterogeneous feature coupling stage, to fully coordinate the classification capabilities of the STM tensor representation and the tensor regularization network, a unified optimization objective is constructed in the preferred embodiment. This objective actively integrates the losses corresponding to reconstruction accuracy (i.e., the loss between the original tensor flow and the reconstructed tensor flow), low-rank gradient domain constraints, Tikhonov regularization, supervised classification, and tensor consistency constraints.

[0096]

[0097] in, This represents the loss between the original tensor flow and the reconstructed tensor flow, i.e., the loss term corresponding to the difference between the higher-order tensor representation of the hyperspectral data and the expanded tensor form. This represents the predicted classification value obtained through the classifier layer; This represents the loss corresponding to the low-rank constraint in the gradient domain, which is the loss term corresponding to the difference between the product of the gradient operator of each mode and the tensor factor of that mode and the product of the low-rank matrix of that mode and the Tikhonov matrix. This represents the loss corresponding to Tikhonov regularization; This represents the loss corresponding to the tensor consistency constraint, that is, the loss term corresponding to the difference between the expanded tensor form and the feature fusion result; Let represent the loss term corresponding to the true label of the grain grain sample image and the label predicted by the recognition model. To explicitly guide feature extraction for classification, a supervised classification loss based on labeled samples is incorporated. The predicted classification probability is obtained by the classifier layer of the recognition model. If the true label is the grain grain sample image, then the classification loss can be specifically expressed as:

[0098]

[0099] Where M represents the total number of samples.

[0100] As can be seen, STM actively utilizes adaptive fully connected tensor modeling to capture comprehensive modal correlations beyond the limitations of traditional neighboring modes, thus significantly enhancing representational capabilities. It actively models spatial continuity and spectral smoothness through novel low-rank gradient domain constraints, greatly reducing redundancy and improving robustness to spectral noise. Simultaneously, combined with Tikhonov regularization, it actively improves numerical stability and generalization performance, proving particularly effective under conditions of limited labeled data. Based on these designs, STM can efficiently extract spectral-spatial features and achieve excellent classification accuracy, demonstrating great application potential in grain quality detection and related hyperspectral image classification tasks. This end-to-end heterogeneous collaborative training of spectral features ensures that the tensor regularization network can learn robust, efficient, and highly discriminative tensor features. This training strategy not only maintains STM's powerful spectral-spatial representation capabilities but also optimizes the computational efficiency of the tensor regularization network, enabling the network to exhibit excellent performance and adaptability in grain quality analysis and related hyperspectral image classification tasks.

[0101] Additionally, refer to Figure 3 In this embodiment, to optimize the feature acquisition of visible light images, the processing method for visible light image branches includes:

[0102] Different groups of dilated convolutional layers with varying dilation rates are used to process the normalized visible light image. The processing results of each group of dilated convolutional layers are weighted and summed, then processed by the FFN module, and finally fused with the processing results of the spatial state model branch to obtain the output of the visible light image branch. The processing of the spatial state model branch includes processing the normalized visible light image through the spatial state model, then fusing it with the unnormalized visible light image to obtain the processing result of the spatial state model branch.

[0103] The weights when performing a weighted summation of the processing results from each dilated convolutional layer are adaptively adjusted during the training of the recognition model.

[0104] Hyperspectral images inherently possess high-dimensional structures and complex spectral-spatial correlations, making it difficult for traditional tensor modeling methods to fully characterize their multimodal coupling features. To address this challenge and obtain a more discriminative structural representation in the spatial domain, a Mamba-driven spatial feature extraction mechanism is introduced into the model, specifically identifying the visible light image branch within the model. This module leverages selective state updates and long-range dependency modeling capabilities to effectively compensate for the shortcomings of convolution in global modeling, further enhancing the continuity and consistency of spatial structure representation. After completing spatial feature modeling, higher-order tensors can be adaptively applied... The hyperspectral image patch is decomposed into interconnected tensor factors, which further explores fine-grained differences in the spectral domain and overcomes the limitations of traditional tensor decomposition methods in terms of fixed rank, limited representation, and noise sensitivity.

[0105] By designing the spatial feature branch as an adaptive grouped dilated convolution (SGDC) structure within Mamba's selective state-space modeling framework, the model's ability to model local and cross-regional dependencies can be enhanced. SGDC achieves multi-scale context aggregation in the spatial domain through three sets of convolutions with different dilation rates, thereby supplementing Mamba's ability to capture long-range dependencies in sequence modeling and improving the expressive strength of binary activation features. Compared to standard convolution, grouped convolution can significantly reduce parameter size and computational cost, making the spatial modeling process more efficient. In one embodiment, the normalized features of the input... For a normalized visible light image, the embedding process of the SGDC module can be defined as follows:

[0106]

[0107]

[0108] in, This represents a 3×3 grouped dilated convolution with an inflation rate of 2n-1. In this embodiment, n∈(1,2,3). All are adaptive grouping adjustment factors (i.e., the weights when the processing results of each group of dilated convolutional layers are weighted and summed), with values ​​ranging from [value range missing]. ∈(0,1), with a default initial value of 0.5.

[0109] ; ; In this model, Linear Normalization (LN) is used to normalize input features to improve network convergence, while the State-Space Model (SSM) module models sequence dependencies through a selective state transition mechanism, thereby enhancing the global context representation capability. The traditional State-Space Model (SSM) originates from control theory, and it represents the current input sequence x(t) ∈ R through hidden states h(t) ∈ R. Its continuous-time system can be expressed as:

[0110]

[0111] Where, coefficient matrix A∈ Represents the state transition term, with coefficient matrix B∈ Represents the control input terms, with coefficient matrix C∈ Let D ∈ R represent the observed output term, and D ∈ R be the feedback term. In deep learning, these coefficients, after discretization, are typically learnable. For continuous-time systems, the input... In time interval Under the assumption of internal invariance, the zero-order preserver (ZOH) method is used to discretize the continuous system. The discretized system is represented as follows:

[0112]

[0113] in, , However, the parameters of traditional SSMs are time-invariant, which limits their ability to handle sequences with complex dynamic changes. To address this issue, the Selective Scan Spatial State Sequence Model (S6) can be employed. Its core idea is to design key parameters as functions of the input sequence, thereby dynamically adjusting the model structure based on the data. S6 not only improves the ability to characterize complex temporal dynamics but also enhances the modeling effect on global dependencies. Building upon this, Mamba introduces B, C, and discrete proportions of input dependencies, enabling the model to dynamically adjust feature representations and effectively model long-term dependencies. In other embodiments, the imperfect grain identification method of this implementation can also be achieved using the S4 or S8 spatial state models.

[0114] In this embodiment, the method for acquiring hyperspectral images of grain kernels includes:

[0115] Each grain is randomly placed in a grid of a predetermined size and interval, facing the spectrometer in any orientation. These grids are then placed on a moving device illuminated by a light source. Once the moving device starts operating, the reflected light from the grains is acquired in real time via a spectral camera, generating a one-dimensional spatial image and corresponding spectral data. As the moving device moves continuously, the image sequence and spectral curves are continuously recorded. These images are then stitched together to form a three-dimensional data cube containing two-dimensional images and spectral dimensions, serving as the original hyperspectral image. The hyperspectral image of the grain is then obtained based on the original hyperspectral image.

[0116] The processing methods for obtaining hyperspectral images of grains from the original hyperspectral images include standard black and white plate correction.

[0117] In one specific embodiment, refer to Figure 4 The image acquisition system mainly consists of a hyperspectral camera, a visible light camera, a matching lens group, a fixing fixture, a displacement control console, a backlight source, a host computer, and a grain sample (taking corn kernels as an example). The hyperspectral camera and the visible light camera are securely mounted on the fixture support, with their optical axes pointing vertically downwards towards the sample stage. Before the experiment, the corn kernel sample is evenly spread on the stage to ensure a flat and uniform surface, improving subsequent imaging quality and data consistency. By adjusting the displacement platform, the field of view of the spectral and image acquisition systems are strictly aligned, achieving consistent spatial correspondence between the two modal data. During the acquisition process, the hyperspectral imaging module is first activated to scan the sample line by line in the visible to near-infrared band using a push-broom method, acquiring a cube of reflectance spectral data; then, the system switches to the visible light industrial camera to acquire high-resolution color images under the same field of view. To enhance the diversity and adaptability of the data, the sampling process was repeated by randomly adjusting the number of samples and their placement, thereby constructing an imperfect multimodal dataset of seeds with high consistency, high coverage and high reliability, providing a solid data foundation for feature extraction, model training and seed phenotypic recognition.

[0118] In this embodiment, the GaiaSorter system was used to acquire hyperspectral images of all corn kernels during data acquisition. The GaiaSorter system comprises core components such as a hyperspectral imager, a diffuse reflection light source, a working distance adjustment structure, an electric translation stage, an electric lifting stage, and a host computer control unit. The RT17 spectrometer used covers the near-infrared band of 902.4-1745.0 nm, with 512 spectral channels, a spectral resolution of 5 nm, a spectral sampling rate of 1.6 nm, a maximum sample space of 300 mm × 300 mm × 100 mm, 640 × 512 pixels, and a frame rate of 4 s / cube, meeting the requirements for high-precision spectral imaging. To ensure good thermal and temporal stability during imaging, automatic spectral emissivity calibration was performed before acquisition to reduce spectral errors caused by baseline drift.

[0119] To address the requirements of online detection, this imperfect grain identification method employs a gridded acquisition strategy: each corn kernel is randomly placed in a self-made 7×7 grid, each grid measuring 10mm×16mm with a 2mm spacing, facing the spectrometer in any orientation. The tray is then placed on an electrically controlled moving platform or conveyor belt (i.e., a moving device) under illumination. Once the platform is operational, the spectroscopic camera captures the reflected light from the sample in real time, generating a one-dimensional spatial image and corresponding spectral data. As the platform moves continuously, the system continuously records image sequences and spectral curves, ultimately stitching them together into a three-dimensional data cube containing both two-dimensional images and spectral dimensions. This process is repeated for all samples until the entire dataset is acquired. To eliminate the influence of dark current on the original spectrum and normalize the sensor's reflectance values, a standard black-and-white plate calibration method is used, with the following calibration formula:

[0120]

[0121] in, This represents the corrected hyperspectral image. Represents the original hyperspectral image. It is a completely black reference image. It is a completely white reference image.

[0122] Furthermore, the methods for acquiring hyperspectral data of grains used as input to the recognition model include:

[0123] Background removal is performed on the acquired hyperspectral images of grain grains to obtain spectral data containing only the grain region. Then, the image portion corresponding to each grain is separated from the whole image, and all pixel regions containing grains in the separated image portion are defined as the region of interest (ROI). Based on this, the average value of the pixel spectrum in the ROI is calculated as the average spectral feature of the grain to obtain the hyperspectral data of grain grains used as input to the recognition model.

[0124] In a preferred embodiment, to obtain the optimal foreground segmentation effect, the imaging quality of each spectral band was first compared. The 1323.2nm band was found to have the highest foreground-background contrast, and therefore it was used to construct the initial segmentation mask. Subsequently, the image of this band was decomposed using HSB channels to obtain three channels: Brightness, Saturation, and Hue. Further analysis showed that the Hue channel could more effectively separate the grains from the background. Based on this, a threshold segmentation method was used on the Hue channel to remove the background region and generate a grain mask. After obtaining the mask, the binary mask was multiplied pixel-by-pixel with all bands to obtain hyperspectral data without background interference. Next, each corn kernel (grain) was independently segmented from the entire image and cropped into a hyperspectral image patch of size n×m×512, where n and m are the maximum bounding dimensions of the corresponding grain outline. After the above preprocessing, as shown... Figure 5 As shown, both the original hyperspectral image and the background-removed grain spectral data retain 512 effective bands, laying a reliable data foundation for subsequent feature extraction and model training.

[0125] The average spectral characteristics can effectively represent the overall information of a single grain because they integrate all spectral pixels within the grain, reflecting the global heterogeneity of the grain in terms of space and composition. Since the sensor's response is unstable at both ends of the spectrum, noise is mainly distributed in the beginning and end bands of the spectrum. Therefore, in this embodiment, the effective band within the 907-1745nm range is retained, i.e., the beginning and end bands of the spectrum are removed to eliminate high-noise regions. The final hyperspectral data consists of 512 effective spectral channels containing the ROI seed pixel block.

[0126] Before training the recognition model, the visible light image samples also need to be obtained by removing the background from the acquired visible images of grains to obtain visible light images containing only the grain regions; then the image portion corresponding to each grain is separated from the whole image to obtain each visible light image block as a different sample.

[0127] Reference Figure 6After the recognition model is trained, it is evaluated by the performance evaluation module. If the evaluation result is unsatisfactory, the recognition model is retrained. The evaluation methods of the performance evaluation module include:

[0128] The recognition results of the recognition model are evaluated using accuracy, precision, recall, mean precision, and Kappa coefficient as indicators. If the recognition results of the recognition model fail to meet the corresponding indicator thresholds, the evaluation results are deemed unqualified.

[0129] Recall rate is obtained from true positives, true negatives, false positives, and false negatives; precision rate is obtained from true positives and false positives; recall rate is obtained from true positives and false negatives; mean precision is calculated from precision rate and recall rate.

[0130] A true positive is the number of pixels in the sample image where both the label and the recognition result are grain regions; a true negative is the number of pixels in the sample image where both the label and the recognition result are background regions; a false positive is the number of pixels in the sample image where the label is background regions but the recognition result is grain regions; and a false negative is the number of pixels in the sample image where the label is grain regions but the recognition result is background regions.

[0131] Specifically, the evaluation metrics are accuracy (Acc), precision (Pre), recall (Re), mean precision (mAP), and Kappa coefficient. The calculation of each metric is as follows:

[0132]

[0133]

[0134]

[0135] Among them, TP (true positive) represents pixels where both the label and the prediction result are in the seed region, TN (true negative) represents pixels where both are in the background region, and FP (false positive) and FN (false negative) correspond to pixels where the label and the prediction result are inconsistent.

[0136]

[0137] Among them, the average accuracy is .

[0138] The Kappa coefficient is used for consistency testing and can also be used to measure recognition accuracy. Kappa is calculated using the following formula:

[0139]

[0140] The Kappa coefficient ranges from [-1, 1]. When k = 1, it indicates perfect agreement; when k = 0, the agreement is comparable to random identification; and when k < 0, the agreement is worse than random identification. For observational consistency (i.e., overall accuracy), it represents the number of correctly identified samples divided by the total number of samples. The expected probability of random consistency is calculated as follows:

[0141]

[0142] Where m represents the total number of categories, and N represents the total number of samples. This represents the number of true samples in class i, while This represents the number of samples predicted from category i. Generally, the higher the Kappa coefficient of the model, the stronger the consistency between its recognition results and the actual situation, and the better its robustness.

[0143] In one embodiment, refer to Figure 6 The following is an example of the entire process of the imperfect grain identification method:

[0144] The hyperspectral equipment and image acquisition sensor are connected to the host computer via connecting cables to ensure effective transmission and processing of spectral and image data. The optical centers of the hyperspectral camera and the visible light camera are perpendicular to the object's surface, ensuring accurate capture of the effective surface of the kernels during shooting. Supplemental lighting is positioned around the camera on the same plane, ensuring the corn kernels are evenly distributed on the platform, providing clear image contrast. An imperfect corn kernel identification method based on coupled Mamba features of spectral-space heterogeneity is proposed. Corn kernels are dispersed on the platform of the test bench. The host computer opens the acquisition software, calibrates the hyperspectral camera and visible light camera to their optimal shooting states, and then sequentially clicks the acquisition button in the hyperspectral software and the image capture button in the visible light camera to acquire data. Specific implementation steps of this invention:

[0145] S1: Construct an experimental device for image acquisition of corn kernels. The experimental device for image acquisition mainly consists of the following components: a hyperspectral camera, a visible light camera, a fixture, an adjustable platform that can be adjusted in all directions, supplementary lights that surround the camera on the same plane in all directions, and a host computer acquisition software system.

[0146] After preprocessing, the data undergoes spectral feature extraction and image feature extraction, followed by heterogeneous feature coupling, and finally input into the classifier.

[0147] S2: For the hyperspectral data section, spectral data preprocessing involves constructing a segmentation mask using the 1323.2nm band, which offers the highest foreground-background contrast. This mask effectively separates the grains from the background in the Hue channel. The mask is then applied to the full-band hyperspectral data to obtain single-grain hyperspectral samples with no background interference and retaining 512 effective bands. Figure 5 As shown. In the image data section, the raw data is sequentially filtered for noise reduction, and then the effective data is normalized.

[0148] In the data acquisition section, individual corn kernels were randomly placed in a self-made 7×7 grid tray, with each grid measuring 10mm×16mm and spaced 2mm apart, ensuring the kernels faced the spectrometer in any orientation. The tray was placed on an electrically controlled moving platform illuminated by a controlled light source. During the platform's uniform movement, the hyperspectral camera simultaneously acquired the reflected light information from the samples, obtaining spatial images and corresponding spectral data row by row, and stitching them together to form a three-dimensional data cube containing both spatial and spectral information. All samples were acquired using the same procedure, and the raw data underwent dark current correction and reflectance normalization.

[0149] S3: The adaptive grouped dilated convolution module captures multi-scale texture features in visible light images. It combines a dynamic grouping strategy with variable dilation rate modeling, enabling the network to acquire fine-grained structural information of defect regions at different spatial scales, thus overcoming the shortcomings of traditional convolution in modeling complex textures and boundary deformations.

[0150] S4: The preprocessed spectral data is input into the tensor regularization module for spectral feature extraction. By dynamically adjusting the tensor rank and simultaneously modeling local consistency and global constraints, the expressive power of spectral features is enhanced. It should be noted that S3 and S4 can actually be performed simultaneously, and there is no strict order relationship.

[0151] S5: By utilizing the spectral-spatial heterogeneous feature coupling module in the recognition model, the advantageous information of the two modalities is jointly rearranged and fused through the heterogeneous feature coupling strategy, so that the differences in spectral composition and spatial texture structure can complement each other in the feature domain.

[0152] The heterogeneous feature coupling strategy specifically refers to the effective correlation modeling across feature spaces for various types of features (such as spectral features, spatial structure features, and texture features) with different sources, statistical properties, and physical meanings. This is achieved through a unified structured mapping and weighted fusion mechanism, while maintaining the differences between various features. The core objective of this strategy is to establish explicit or implicit interaction relationships between different features in a shared representation space, enabling various features to constrain and complement each other.

[0153] In this embodiment, the specific operation process of jointly rearranging and fusing the advantageous information of the two modalities through the heterogeneous feature coupling strategy includes: First, mapping the two types of features (i.e., the output features of the visible light image branch and the output features of the hyperspectral data branch) to a unified feature dimension space; then, using a learnable weight matrix, weighting the spectral features and spatial features; thus, in the tensor coupling stage, the spectral features are not only related to their own information, but also interact with the spatial morphological features; finally, a comprehensive representation can be formed, enabling the recognition model to simultaneously use "spectral anomalies + specific spatial morphology" to determine the imperfect particle type.

[0154] S6: Performance evaluation module, which evaluates the recognition results and provides an effective assessment in terms of objective metrics. The objective evaluation metrics for model parameters are, in order, overall accuracy (OA), mAP, and Kappa coefficient;

[0155] Experimental results show that, compared with the most advanced existing deep models, the imperfect grain identification method in this embodiment has achieved significant improvements in identification accuracy, robustness, and generalization ability under complex environments. It provides an efficient, reliable, and engineering-application-potential solution for intelligent grain quality detection and automatic identification of imperfect grains.

[0156] Implementation of an Imperfect Grain Identification System Based on Spectral Spatial Coupling Features

[0157] This embodiment provides a technical solution for an imperfect grain identification system based on spectral spatial coupling features, including a processor containing executable program instructions. These instructions are executed by the processor to implement the imperfect grain identification system based on spectral spatial coupling features, specifically including:

[0158] The visible light image and hyperspectral data of the grain are input into the recognition model; the visible light image branch in the recognition model processes the visible light image of the grain to obtain the output of the branch;

[0159] The hyperspectral data branch in the recognition model processes the hyperspectral data of grains to obtain its output; the recognition result is obtained based on the outputs of the visible light image branch and the hyperspectral data branch; the recognition model is trained using grain sample images.

[0160] The processing method of the hyperspectral data branch includes: expanding the hyperspectral data represented by high-order tensors into tensor factors containing spectral and spatial information features of various modes, and multiplying them together, with each mode having a tensor form that superimposes all different spectral information features; and extracting features from the expanded tensor form to obtain the output of this branch.

[0161] The loss function during model training includes loss terms corresponding to the true labels of grain sample images and the labels predicted by the recognition model, and loss terms corresponding to the difference between the high-order tensor representation and the expanded tensor form of the hyperspectral data.

[0162] Therefore, by dynamically adjusting the effective rank structure of the spectral tensor through tensor expansion, the feature representation capability is significantly enhanced, and a more comprehensive intermodal correlation can be captured. Adaptive fully connected tensor modeling can capture comprehensive modal correlations beyond the limitations of traditional adjacent modes, thus significantly enhancing representation capability. On this basis, by fully coordinating the tensor representation and the overall classification capability through the loss in the tensor expansion process and the loss in the classification recognition result, the recognition accuracy of the recognition model is improved. Hyperspectral data is essentially a high-order tensor (e.g., space × space × spectrum). The spatial dimension contains local structure, texture, and morphological information; the spectral dimension contains material composition and spectral correlation information. There is a high degree of correlation and redundancy between different spectral bands. Expanding the high-order tensor into the multiplication of tensor factors on multiple modes is essentially decomposing the "strongly coupled high-dimensional representation" into a "low-dimensional factor representation of multi-subspace collaborative modeling." This low-dimensional factor representation naturally possesses the characteristics of controllable structure, reconfigurable information, and adjustable rank. Furthermore, since each mode of hyperspectral data expansion superimposes all different spectral information features, the effective rank is not fixed but adaptively changes with data distribution and the optimization process. This means that the effective rank structure of the spectral tensor can be dynamically adjusted during training. In the expanded form, the factor matrix of each mode corresponds to the global linear mapping under that mode; each column of the factor matrix interacts with all spectral / spatial features. In other words, any spectral dimension feature can be associated with any spatial location through the tensor kernel. Therefore, the operation of expanding hyperspectral data represented by higher-order tensors is functionally equivalent to the global receptive field of a fully connected layer.

[0163] Hyperspectral images inherently possess high-dimensional structures and complex spectral-spatial correlations, making it difficult for traditional tensor modeling methods to fully characterize their multimodal coupling features. To address this challenge, obtain more discriminative structural representations in the spatial domain, further explore fine-grained differences in the spectral domain, and overcome the limitations of traditional tensor decomposition methods in terms of fixed rank, limited representation, and noise sensitivity, a preferred embodiment adaptively transforms high-order tensors... The hyperspectral image patch (i.e., hyperspectral data represented by a higher-order tensor) is decomposed into interconnected tensor factors, and the correlations between complex modes are explicitly captured through an adaptive fully connected structure. The expanded tensor form is then obtained. The specific expression is as follows:

[0164]

[0165] The reconstructed tensor features encapsulate the underlying spectral-spatial information. During model training, the recognition model adaptively learns tensor factors associated with each pattern. It actively captures the inherent structural relationships without relying on traditional fully connected tensor decomposition methods.

[0166] To more effectively model the local spatial continuity and spectral smoothness of hyperspectral images, this embodiment proactively introduces an innovative gradient domain low-rank constraint on the tensor factor of the mode expansion. Specifically, the gradient domain low-rank constraint is applied to the expanded tensor form, and the constraint condition is that the product of the gradient operator along each mode and the tensor factor of that mode is equal to the product of the low-rank matrix of that mode and the Tikhonov matrix.

[0167] The loss function during model training also includes a loss term corresponding to the difference between the product of the gradient operator and the tensor factor of each pattern and the product of the low-rank matrix and the Tikhonov matrix of that pattern. Specifically, for the expanded matrix... The low-rank constraint in the gradient domain can be expressed as:

[0168]

[0169] in, It is the gradient operator along the k-th mode, used to capture local spatial continuity and spectral correlation. Low-rank matrix. With Tikhonov matrix It is used to actively reduce redundant information while enhancing the robustness of feature extraction.

[0170] To further stabilize the adaptive tensor decomposition and prevent overfitting when labeled data is limited, this embodiment also introduces Tikhonov regularization during the training of the recognition model. Specifically, the loss function during the training of the recognition model also includes a Tikhonov regularization loss term, which is obtained by summing the Tikhonov regularization results corresponding to each mode. The Tikhonov regularization results corresponding to each mode are obtained by weighted summation of the results of taking the L2 norm of the low-rank matrix, the Tikhonov matrix, and the tensor factor, respectively.

[0171] In a preferred embodiment, the Frobenius norm is specifically used, and the Tikhonov regularization result for each mode is... Specifically as follows:

[0172]

[0173] in Let Frobenius norm be denoted, and N be the total number of patterns. All of these are regularization factors. This regularization actively controls the complexity of the recognition model, improves numerical stability, and significantly enhances the recognition model's ability to generalize to unseen data.

[0174] Furthermore, in this embodiment, the method for feature extraction of the aforementioned tensor form includes:

[0175] The expanded tensor form is processed by 3D convolution path and 2D convolution path respectively, and the features output by 3D and 2D convolution path are fused. Based on the feature fusion result, the output of the hyperspectral data branch is obtained.

[0176] The processing method for 3D convolution paths includes performing 3D convolution on the expanded tensor form mentioned above, and superimposing the constant term of the 3D convolution path.

[0177] The processing method for 2D convolution paths includes performing 2D convolution on the dimensionality reconstruction result of the expanded tensor form mentioned above, and superimposing the constant term of the 2D convolution path.

[0178] Based on this, and considering the limited computing resources in actual grain quality testing, this embodiment adopts a lightweight design strategy. Therefore, the output of the hyperspectral data branch is obtained according to the feature fusion results in the following ways:

[0179] The feature fusion result is subjected to a depthwise separable convolution to obtain depthwise separable features. The result of processing the depthwise separable features through a channel attention mechanism is then fused with the depthwise separable features, and the output of the hyperspectral data branch is obtained based on the fusion result. These optimizations actively reduce computational complexity and memory consumption, enabling the tensor regularized network to efficiently adapt to the real-time inference requirements of edge devices.

[0180] In a preferred embodiment, the tensor regularization network of the hyperspectral data branch is designed as an end-to-end jointly optimized architecture, enabling heterogeneous training of tensor decomposition and feature extraction of visible light images.

[0181] Specifically, the tensor regularization network employs a dual convolutional path to directly utilize the unfolded tensor features for efficient feature integration. The 3D convolutional path directly processes the provided tensor features to preserve the complete spectral-spatial structural information. This is expressed as follows:

[0182]

[0183] 2D convolutional paths primarily process the spatial dimension of tensor features, actively extracting local spatial patterns and structural information, as specifically expressed below:

[0184]

[0185] In a tensor regularized network (i.e., an STM model), the outputs of 3D and 2D convolutional paths are fused, as specifically expressed below:

[0186]

[0187] In the depthwise separable convolution path, which includes depthwise separable convolution and channel attention mechanisms, standard convolution is decomposed into depthwise separable convolution. and channel attention mechanism In a two-step operation, the hyperspectral data branch network of the recognition model significantly reduces the number of parameters and computational cost while maintaining feature extraction capabilities. Such depthwise separable convolutional paths can be expressed as follows:

[0188]

[0189]

[0190]

[0191] but This is the output of the STM Model. The output of the hyperspectral data branch can be obtained from this output through residual connection.

[0192] In this embodiment, the loss function during model training also includes a loss term corresponding to the difference between the expanded tensor form and the feature fusion result. In summary, during the heterogeneous feature coupling stage, to fully coordinate the classification capabilities of the STM tensor representation and the tensor regularization network, a unified optimization objective is constructed in the preferred embodiment. This objective actively integrates the losses corresponding to reconstruction accuracy (i.e., the loss between the original tensor flow and the reconstructed tensor flow), low-rank gradient domain constraints, Tikhonov regularization, supervised classification, and tensor consistency constraints.

[0193]

[0194] in, This represents the loss between the original tensor flow and the reconstructed tensor flow, i.e., the loss term corresponding to the difference between the higher-order tensor representation of the hyperspectral data and the expanded tensor form. This represents the predicted classification value obtained through the classifier layer; This represents the loss corresponding to the low-rank constraint in the gradient domain, which is the loss term corresponding to the difference between the product of the gradient operator of each mode and the tensor factor of that mode and the product of the low-rank matrix of that mode and the Tikhonov matrix. This represents the loss corresponding to Tikhonov regularization; This represents the loss corresponding to the tensor consistency constraint, that is, the loss term corresponding to the difference between the expanded tensor form and the feature fusion result; Let represent the loss term corresponding to the true label of the grain grain sample image and the label predicted by the recognition model. To explicitly guide feature extraction for classification, a supervised classification loss based on labeled samples is incorporated. The predicted classification probability is obtained by the classifier layer of the recognition model. If the true label is the grain grain sample image, then the classification loss can be specifically expressed as:

[0195]

[0196] Where M represents the total number of samples.

[0197] As can be seen, STM actively utilizes adaptive fully connected tensor modeling to capture comprehensive modal correlations beyond the limitations of traditional neighboring modes, thus significantly enhancing representational capabilities. It actively models spatial continuity and spectral smoothness through novel low-rank gradient domain constraints, greatly reducing redundancy and improving robustness to spectral noise. Simultaneously, combined with Tikhonov regularization, it actively improves numerical stability and generalization performance, proving particularly effective under conditions of limited labeled data. Based on these designs, STM can efficiently extract spectral-spatial features and achieve excellent classification accuracy, demonstrating great application potential in grain quality detection and related hyperspectral image classification tasks. This end-to-end heterogeneous collaborative training of spectral features ensures that the tensor regularization network can learn robust, efficient, and highly discriminative tensor features. This training strategy not only maintains STM's powerful spectral-spatial representation capabilities but also optimizes the computational efficiency of the tensor regularization network, enabling the network to exhibit excellent performance and adaptability in grain quality analysis and related hyperspectral image classification tasks.

[0198] In addition, in this embodiment, to optimize the feature acquisition of visible light images, the processing method for visible light image branches includes:

[0199] Different groups of dilated convolutional layers with varying dilation rates are used to process the normalized visible light image. The processing results of each group of dilated convolutional layers are weighted and summed, then processed by the FFN module, and finally fused with the processing results of the spatial state model branch to obtain the output of the visible light image branch. The processing of the spatial state model branch includes processing the normalized visible light image through the spatial state model, then fusing it with the unnormalized visible light image to obtain the processing result of the spatial state model branch.

[0200] The weights when performing a weighted summation of the processing results from each dilated convolutional layer are adaptively adjusted during the training of the recognition model.

[0201] Hyperspectral images inherently possess high-dimensional structures and complex spectral-spatial correlations, making it difficult for traditional tensor modeling methods to fully characterize their multimodal coupling features. To address this challenge and obtain a more discriminative structural representation in the spatial domain, a Mamba-driven spatial feature extraction mechanism is introduced into the model, specifically identifying the visible light image branch within the model. This module leverages selective state updates and long-range dependency modeling capabilities to effectively compensate for the shortcomings of convolution in global modeling, further enhancing the continuity and consistency of spatial structure representation. After completing spatial feature modeling, higher-order tensors can be adaptively applied... The hyperspectral image patch is decomposed into interconnected tensor factors, which further explores fine-grained differences in the spectral domain and overcomes the limitations of traditional tensor decomposition methods in terms of fixed rank, limited representation, and noise sensitivity.

[0202] By designing the spatial feature branch as an adaptive grouped dilated convolution (SGDC) structure within Mamba's selective state-space modeling framework, the model's ability to model local and cross-regional dependencies can be enhanced. SGDC achieves multi-scale context aggregation in the spatial domain through three sets of convolutions with different dilation rates, thereby supplementing Mamba's ability to capture long-range dependencies in sequence modeling and improving the expressive strength of binary activation features. Compared to standard convolution, grouped convolution can significantly reduce parameter size and computational cost, making the spatial modeling process more efficient. In one embodiment, the normalized features of the input... For a normalized visible light image, the embedding process of the SGDC module can be defined as follows:

[0203]

[0204]

[0205] in, This represents a 3×3 grouped dilated convolution with an inflation rate of 2n-1. In this embodiment, n∈(1,2,3). All are adaptive grouping adjustment factors (i.e., the weights when the processing results of each group of dilated convolutional layers are weighted and summed), with values ​​ranging from [value range missing]. ∈(0,1), with a default initial value of 0.5.

[0206]

[0207]

[0208] In this model, Linear Normalization (LN) is used to normalize input features to improve network convergence, while the State-Space Model (SSM) module models sequence dependencies through a selective state transition mechanism, thereby enhancing the global context representation capability. The traditional State-Space Model (SSM) originates from control theory, and it represents the current input sequence x(t) ∈ R through hidden states h(t) ∈ R. Its continuous-time system can be expressed as:

[0209]

[0210] Where, coefficient matrix A∈ Represents the state transition term, with coefficient matrix B∈ Represents the control input terms, with coefficient matrix C∈ Let D ∈ R represent the observed output term, and D ∈ R be the feedback term. In deep learning, these coefficients, after discretization, are typically learnable. For continuous-time systems, the input... Under the assumption that the system remains invariant over a time interval Δt, the zero-order preserver (ZOH) method is used to discretize the continuous system. The discretized system is represented as follows:

[0211]

[0212] in, , However, the parameters of traditional SSMs are time-invariant, which limits their ability to handle sequences with complex dynamic changes. To address this issue, the Selective Scan Spatial State Sequence Model (S6) can be employed. Its core idea is to design key parameters as functions of the input sequence, thereby dynamically adjusting the model structure based on the data. S6 not only improves the ability to characterize complex temporal dynamics but also enhances the modeling effect on global dependencies. Building upon this, Mamba introduces B, C, and discrete proportions of input dependencies, enabling the model to dynamically adjust feature representations and effectively model long-term dependencies. In other embodiments, the S4 or S8 spatial state models can also achieve the imperfect grain identification system of this implementation.

[0213] In this embodiment, the method for acquiring hyperspectral images of grain kernels includes:

[0214] Each grain is randomly placed in a grid of a predetermined size and interval, facing the spectrometer in any orientation. These grids are then placed on a moving device illuminated by a light source. Once the moving device starts operating, the reflected light from the grains is acquired in real time via a spectral camera, generating a one-dimensional spatial image and corresponding spectral data. As the moving device moves continuously, the image sequence and spectral curves are continuously recorded. These images are then stitched together to form a three-dimensional data cube containing two-dimensional images and spectral dimensions, serving as the original hyperspectral image. The hyperspectral image of the grain is then obtained based on the original hyperspectral image.

[0215] The processing methods for obtaining hyperspectral images of grains from the original hyperspectral images include standard black and white plate correction.

[0216] In one specific embodiment, the image acquisition system mainly consists of a hyperspectral camera, a visible light camera, a matching lens group, a fixing fixture, a displacement control console, a backlight source, a host computer, and a grain sample (taking corn kernels as an example). The hyperspectral camera and the visible light camera are securely mounted on the fixture support, with their optical axes pointing vertically downwards towards the sample stage. Before the experiment, the corn kernel sample is evenly spread on the stage to ensure a flat and uniform surface, improving subsequent imaging quality and data consistency. By adjusting the displacement platform, the spectral and image acquisition system's field of view are strictly aligned, achieving consistent spatial correspondence between the two modal data. During the acquisition process, the hyperspectral imaging module is first activated to scan the sample line by line in the visible to near-infrared band using a push-broom method, acquiring a cube of reflectance spectral data; then, the system switches to the visible light industrial camera to acquire high-resolution color images under the same field of view. To enhance the diversity and adaptability of the data, the sampling process was repeated by randomly adjusting the number of samples and their placement, thereby constructing an imperfect multimodal dataset of seeds with high consistency, high coverage and high reliability, providing a solid data foundation for feature extraction, model training and seed phenotypic recognition.

[0217] In this embodiment, the GaiaSorter system was used to acquire hyperspectral images of all corn kernels during data acquisition. The GaiaSorter system comprises core components such as a hyperspectral imager, a diffuse reflection light source, a working distance adjustment structure, an electric translation stage, an electric lifting stage, and a host computer control unit. The RT17 spectrometer used covers the near-infrared band of 902.4-1745.0 nm, with 512 spectral channels, a spectral resolution of 5 nm, a spectral sampling rate of 1.6 nm, a maximum sample space of 300 mm × 300 mm × 100 mm, 640 × 512 pixels, and a frame rate of 4 s / cube, meeting the requirements for high-precision spectral imaging. To ensure good thermal and temporal stability during imaging, automatic spectral emissivity calibration was performed before acquisition to reduce spectral errors caused by baseline drift.

[0218] To meet the requirements of online detection, this imperfect grain identification system designed and implemented a gridded acquisition strategy: each corn kernel was randomly placed in a self-made 7×7 grid, each grid measuring 10mm×16mm with a grid spacing of 2mm, and positioned so that it faced the spectrometer in any orientation. The tray was then placed on an electrically controlled moving platform or conveyor belt (i.e., a moving device) under light illumination. Once the platform started operating, the spectroscopic camera captured the reflected light from the sample in real time through its lens, generating a one-dimensional spatial image and corresponding spectral data. As the platform moved continuously, the system continuously recorded image sequences and spectral curves, ultimately stitching them together into a three-dimensional data cube containing two-dimensional images and spectral dimensions. This process was repeated for all samples until the entire dataset was acquired. To eliminate the influence of dark current on the original spectrum and to normalize the sensor's reflectance values, a standard black and white card calibration method was used, with the following calibration formula:

[0219]

[0220] in, This represents the corrected hyperspectral image. Represents the original hyperspectral image. It is a completely black reference image. It is a completely white reference image.

[0221] Furthermore, the methods for acquiring hyperspectral data of grains used as input to the recognition model include:

[0222] Background removal is performed on the acquired hyperspectral images of grain grains to obtain spectral data containing only the grain region. Then, the image portion corresponding to each grain is separated from the whole image, and all pixel regions containing grains in the separated image portion are defined as the region of interest (ROI). Based on this, the average value of the pixel spectrum in the ROI is calculated as the average spectral feature of the grain to obtain the hyperspectral data of grain grains used as input to the recognition model.

[0223] In a preferred embodiment, to obtain the optimal foreground segmentation effect, the imaging quality of each spectral band was first compared. The 1323.2nm band was found to have the highest foreground-background contrast, and therefore it was used to construct the initial segmentation mask. Subsequently, the image of this band was decomposed using HSB channels to obtain three channels: Brightness, Saturation, and Hue. Further analysis showed that the Hue channel could more effectively separate the grains from the background. Based on this, a threshold segmentation method was applied to the Hue channel to remove the background region and generate a grain mask. After obtaining the mask, the binary mask was multiplied pixel-by-pixel with all bands to obtain hyperspectral data free of background interference. Next, each corn kernel (grain) was independently segmented from the entire image and cropped into a hyperspectral image patch of size n×m×512, where n and m are the maximum bounding dimensions of the corresponding grain outline. After the above preprocessing, both the original hyperspectral image and the background-removed grain spectral data retained 512 effective bands, laying a reliable data foundation for subsequent feature extraction and model training.

[0224] The average spectral characteristics can effectively represent the overall information of a single grain because they integrate all spectral pixels within the grain, reflecting the global heterogeneity of the grain in terms of space and composition. Since the sensor's response is unstable at both ends of the spectrum, noise is mainly distributed in the beginning and end bands of the spectrum. Therefore, in this embodiment, the effective band within the 907-1745nm range is retained, i.e., the beginning and end bands of the spectrum are removed to eliminate high-noise regions. The final hyperspectral data consists of 512 effective spectral channels containing the ROI seed pixel block.

[0225] Before training the recognition model, the visible light image samples also need to be obtained by removing the background from the acquired visible images of grains to obtain visible light images containing only the grain regions; then the image portion corresponding to each grain is separated from the whole image to obtain each visible light image block as a different sample.

[0226] After the recognition model is trained, it is evaluated by the performance evaluation module. If the evaluation result is unsatisfactory, the recognition model is retrained. The evaluation methods of the performance evaluation module include:

[0227] The recognition results of the recognition model are evaluated using accuracy, precision, recall, mean precision, and Kappa coefficient as indicators. If the recognition results of the recognition model fail to meet the corresponding indicator thresholds, the evaluation results are deemed unqualified.

[0228] In one embodiment, refer to Figure 6 The following is an example of the entire process of the imperfect grain identification method:

[0229] The hyperspectral equipment and image acquisition sensor are connected to the host computer via connecting cables to ensure effective transmission and processing of spectral and image data. The optical centers of the hyperspectral camera and the visible light camera are perpendicular to the object's surface, ensuring accurate capture of the effective surface of the kernels during shooting. Supplemental lighting is positioned around the camera on the same plane, ensuring the corn kernels are evenly distributed on the platform, providing clear image contrast. An imperfect corn kernel identification method based on coupled Mamba features of spectral-space heterogeneity is proposed. Corn kernels are dispersed on the platform of the test bench. The host computer opens the acquisition software, calibrates the hyperspectral camera and visible light camera to their optimal shooting states, and then sequentially clicks the acquisition button in the hyperspectral software and the image capture button in the visible light camera to acquire data. Specific implementation steps of this invention:

[0230] STEP 1: Construct an experimental device for image acquisition of corn kernels. The experimental device mainly consists of the following components: a hyperspectral camera, a visible light camera, a fixture, an adjustable platform that can be adjusted in all directions, supplementary lights that surround the camera on the same plane in all directions, and a host computer acquisition software system.

[0231] After preprocessing, the data undergoes spectral feature extraction and image feature extraction, followed by heterogeneous feature coupling, and finally input into the classifier.

[0232] STEP 2: Hyperspectral data section. Preprocessing of the spectral data involves constructing a segmentation mask using the 1323.2nm band with the highest foreground-background contrast. This mask effectively separates the grains from the background in the Hue channel. The mask is then applied to the full-band hyperspectral data to obtain single-grain hyperspectral samples with no background interference and retaining 512 effective bands. Figure 5 As shown. In the image data section, the raw data is sequentially filtered for noise reduction, and then the effective data is normalized.

[0233] In the data acquisition section, individual corn kernels were randomly placed in a self-made 7×7 grid tray, with each grid measuring 10mm×16mm and spaced 2mm apart, ensuring the kernels faced the spectrometer in any orientation. The tray was placed on an electrically controlled moving platform illuminated by a controlled light source. During the platform's uniform movement, the hyperspectral camera simultaneously acquired the reflected light information from the samples, obtaining spatial images and corresponding spectral data row by row, and stitching them together to form a three-dimensional data cube containing both spatial and spectral information. All samples were acquired using the same procedure, and the raw data underwent dark current correction and reflectance normalization.

[0234] STEP 3: The adaptive grouped dilated convolution module captures multi-scale texture features in visible light images. It combines a dynamic grouping strategy with variable dilation rate modeling, enabling the network to acquire fine-grained structural information of defect regions at different spatial scales, thus overcoming the shortcomings of traditional convolution in modeling complex textures and boundary deformations.

[0235] STEP4: The preprocessed spectral data is input into the tensor regularization module for spectral feature extraction. By dynamically adjusting the tensor rank and simultaneously modeling local consistency and global constraints, the expressive power of spectral features is enhanced. It should be noted that STEP3 and STEP4 can actually be performed simultaneously, and there is no strict order relationship.

[0236] STEP5: Utilize the spectral-spatial heterogeneous feature coupling module in the recognition model to jointly rearrange and fuse the advantageous information of the two modalities through a heterogeneous feature coupling strategy, so that the differences in spectral composition and spatial texture structure can complement each other in the feature domain.

[0237] The structural feature coupling strategy specifically refers to the effective correlation modeling across feature spaces for various types of features (such as spectral features, spatial structure features, and texture features) with different sources, statistical properties, and physical meanings. This is achieved through a unified structured mapping and weighted fusion mechanism, while maintaining the differences between various features. The core objective of this strategy is to establish explicit or implicit interaction relationships between different features in a shared representation space, enabling various features to constrain and complement each other.

[0238] In this embodiment, the specific operation process of jointly rearranging and fusing the advantageous information of the two modalities through the heterogeneous feature coupling strategy includes: First, mapping the two types of features (i.e., the output features of the visible light image branch and the output features of the hyperspectral data branch) to a unified feature dimension space; then, using a learnable weight matrix, weighting the spectral features and spatial features; thus, in the tensor coupling stage, the spectral features are not only related to their own information, but also interact with the spatial morphological features; finally, a comprehensive representation can be formed, enabling the recognition model to simultaneously use "spectral anomalies + specific spatial morphology" to determine the imperfect particle type.

[0239] STEP 6: Performance Evaluation Module. This module evaluates the recognition results and provides an effective assessment based on objective metrics. The objective evaluation metrics for model parameters are, in order, overall accuracy (OA), mAP, and Kappa coefficient.

[0240] Experimental results show that, compared with the most advanced existing deep models, the imperfect grain identification method in this embodiment has achieved significant improvements in identification accuracy, robustness, and generalization ability under complex environments. It provides an efficient, reliable, and engineering-application-potential solution for intelligent grain quality detection and automatic identification of imperfect grains.

[0241] It should be understood that the above-described specific embodiments of the present invention are merely illustrative or explanatory of the principles of the present invention, and do not constitute a limitation thereof.

Claims

1. An imperfect method for identifying grain kernels based on spectral spatial coupling features, characterized in that, include: The visible light image and hyperspectral data of the grain are input into the recognition model; the visible light image branch in the recognition model processes the visible light image of the grain to obtain the output of the branch; The hyperspectral data branch in the recognition model processes the hyperspectral data of grains to obtain its output; the recognition result is obtained based on the outputs of the visible light image branch and the hyperspectral data branch; the recognition model is trained using grain sample images. The processing method of the hyperspectral data branch includes: expanding the hyperspectral data represented by high-order tensors into tensor factors containing spectral and spatial information features of various modes, and multiplying them together, with each mode having a tensor form that superimposes all different spectral information features; and performing feature extraction on the tensor form to obtain the output of the branch. The loss function during model training includes loss terms corresponding to the true labels of grain sample images and the labels predicted by the recognition model, and loss terms corresponding to the difference between the high-order tensor representation of the hyperspectral data and the tensor form.

2. The imperfect grain identification method based on spectral spatial coupling features according to claim 1, characterized in that, The processing methods for visible light image branches include: Different groups of dilated convolutional layers with varying dilation rates are used to process the normalized visible light image. The processing results of each group of dilated convolutional layers are weighted and summed, then processed by the FFN module, and finally fused with the processing results of the spatial state model branch to obtain the output of the visible light image branch. The processing of the spatial state model branch includes processing the normalized visible light image through the spatial state model, then fusing it with the unnormalized visible light image, and finally obtaining the processing result of the spatial state model branch based on the fusion result. The weights when performing a weighted summation of the processing results from each dilated convolutional layer are adaptively adjusted during the training of the recognition model.

3. The imperfect grain identification method based on spectral spatial coupling features according to claim 1 or 2, characterized in that, The expanded tensor form is subjected to a low-rank constraint in the gradient domain, wherein the product of the gradient operator along each mode and the tensor factor of that mode is equal to the product of the low-rank matrix of that mode and the Tikhonov matrix. The loss function during model training also includes a loss term corresponding to the difference between the product of the gradient operator of each pattern and the tensor factor of that pattern and the product of the low-rank matrix of that pattern and the Tikhonov matrix.

4. The imperfect grain identification method based on spectral spatial coupling features according to claim 1 or 2, characterized in that, The loss function during model training also includes a Tikhonov regularization loss term, which is obtained by summing the Tikhonov regularization results for each mode. The Tikhonov regularization results for each mode are obtained by weighted summing the results of taking the L2 norm of the low-rank matrix, the Tikhonov matrix, and the tensor factor respectively.

5. The imperfect grain identification method based on spectral spatial coupling features according to claim 1 or 2, characterized in that, The methods for feature extraction from the tensor form include: The tensor form is processed by 3D convolutional paths and 2D convolutional paths respectively, and the features output by the 3D and 2D convolutional paths are fused. Based on the feature fusion result, the output of the hyperspectral data branch is obtained. The processing method of the 3D convolution path includes performing 3D convolution processing on the tensor form and superimposing the constant term of the 3D convolution path; The processing method of the 2D convolution path includes performing 2D convolution processing on the dimensionality reconstruction result in the tensor form, and superimposing the constant term of the 2D convolution path.

6. The imperfect grain identification method based on spectral spatial coupling features according to claim 5, characterized in that, The loss function during model training also includes a loss term corresponding to the difference between the expanded tensor form and the feature fusion result.

7. The imperfect grain identification method based on spectral spatial coupling features according to claim 1 or 2, characterized in that, Methods for acquiring hyperspectral data of grains used as input for the recognition model include: Background removal is performed on the acquired hyperspectral images of grain grains to obtain spectral data containing only the grain regions. Then, the image portion corresponding to each grain is separated from the whole image, and all pixel regions containing grains in the separated image portion are defined as regions of interest. Based on this, the average value of the pixel spectrum in the region of interest is calculated as the average spectral feature of the grain to obtain the hyperspectral data of grain grains used as input to the recognition model.

8. The imperfect grain identification method based on spectral spatial coupling features according to claim 7, characterized in that, Methods for acquiring hyperspectral images of grain grains include: Each grain is randomly placed in a grid of a predetermined size and interval, facing the spectrometer in any orientation. The grid is then placed on a moving device illuminated by a light source. Once the moving device starts operating, the reflected light from the grain is acquired in real time by a spectral camera, generating a one-dimensional spatial image and corresponding spectral data. As the moving device moves continuously, the image sequence and spectral curve are continuously recorded. These are then stitched together to form a three-dimensional data cube containing a two-dimensional image and a spectral dimension, serving as the original hyperspectral image. The hyperspectral image of the grain is obtained based on the original hyperspectral image. The processing methods for obtaining hyperspectral images of grains from the original hyperspectral images include standard black and white plate correction.

9. The imperfect grain identification method based on spectral spatial coupling features according to claim 5, characterized in that, Based on the feature fusion results, the output of the hyperspectral data branch can be obtained in the following ways: The feature fusion result is subjected to depthwise separable convolution to obtain depthwise separable features; the result of the depthwise separable features after being processed by the channel attention mechanism is then fused with the depthwise separable features, and the output of the hyperspectral data branch is obtained based on the fusion result.

10. An imperfect grain identification system based on spectral spatial coupling features, comprising a processor, wherein the processor stores executable program instructions, characterized in that, The executable program instructions are executed by a processor to implement the imperfect grain identification method based on spectral spatial coupling features as described in any one of claims 1-9.

Images