Broccoli GRA grading detection method based on multi-modal fusion and spectral quantification

Abstract

The application discloses a broccoli GRA grading detection method based on multi-modal fusion and spectral quantification, and a two-stage grading detection architecture of rapid nondestructive preliminary screening-targeted accurate quantification is constructed, multi-modal feature fusion of ordinary RGB images is used to realize rapid nondestructive preliminary screening of a large number of samples, and the efficiency of the large number of sample detection is improved; a cross-modal attention fusion mechanism of morphological features and deep visual features is constructed, and an effective mapping relationship between appearance visual features and internal GRA content is established; for the samples reserved by preliminary screening, targeted spectrum collection and core feature wavelength screening are combined, a targeted collection area is positioned through visual features, the pertinence and effectiveness of spectrum collection are improved, the feature dimension of spectrum is reduced through feature wavelength screening, and the calculation amount and time consumption of spectrum detection are reduced; and a CNN-PLSR hybrid algorithm is combined, the nonlinear representation ability of spectrum features and the stability of linear regression are considered, and the detection precision is improved.

Description

Technical Field

[0001] This invention relates to the field of non-destructive testing in smart agriculture, specifically to a GRA grading and testing method for broccoli based on multimodal fusion and spectral quantification. Background Technology

[0002] Currently, the detection technologies for GRA content in broccoli are mainly divided into the following categories, all of which have a core deficiency that makes them unsuitable for large-scale field applications: The first category is traditional laboratory chemical detection methods. The detection process requires destructive pretreatment of the sample, such as freeze-drying, grinding, and multi-step solvent extraction. This causes irreversible damage to the plant and makes it impossible to detect live organisms. The second category is spectral non-destructive testing technology, which is time-consuming and inefficient, making it difficult to deploy routinely in ordinary planting bases; adopting the full sample spectral detection mode cannot reduce the cost and time consumption of batch testing; The third category is crop phenotypic detection technology based on computer vision. Existing related technologies are mostly focused on grading the appearance maturity of broccoli, identifying pests and diseases, and predicting yield. Research on RGB visual detection of GRA biochemical components inside broccoli is currently lacking.

[0003] Existing technologies cannot simultaneously meet the requirements of low cost, high efficiency, and high precision in GRA detection. Therefore, it is necessary to design a grading and detection method for broccoli GRA based on multimodal fusion and spectral quantification to achieve non-destructive, low-cost, high-efficiency, and high-precision grading and detection of broccoli GRA content in field conditions, thereby solving the problem that existing technologies cannot be adapted to large-scale field applications. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a grading detection method for broccoli GRA based on multimodal fusion and spectral quantification. By constructing a two-level grading detection architecture of rapid and non-destructive initial screening and targeted and precise quantification, rapid and non-destructive initial screening of a large number of samples is achieved through multimodal feature fusion of ordinary RGB images. Then, by collecting spectral data and using a CNN-PLSR hybrid algorithm for detection, the detection accuracy is improved.

[0005] The technical solution to achieve the objective of this invention is as follows: A method for GRA grading and detection of broccoli based on multimodal fusion and spectral quantification includes: RGB images of functional leaves of broccoli were acquired. The RGB images of functional leaves of broccoli were preprocessed using bilateral filtering algorithm and multi-scale MSR algorithm to obtain enhanced RGB images. An attention mechanism was introduced into the YOLOv8 segmentation network to extract the region of interest (ROI) image from the enhanced RGB image. A dual-branch feature extraction architecture is constructed, consisting of morphological coding feature extraction and deep visual coding feature extraction. Morphological coding features are extracted from ROI images through the morphological coding feature extraction branch, and deep visual coding features are extracted from ROI images through the deep visual coding feature extraction branch. A cross-modal attention fusion module is used to achieve adaptive complementary fusion of the two types of features to obtain multimodal features, which are then input into a binary classification screening model for GRA content to perform initial screening of broccoli and obtain samples to be refined. For the samples retained from the initial screening, based on the multimodal features extracted during the initial screening process, the representative region in the leaves with the highest correlation to GRA content is located as the target region for spectral acquisition. Spectral data is acquired from the target region. A continuous projection algorithm is used to project and transform the spectral data to obtain low collinearity characteristic wavelengths. A competitive adaptive reweighting algorithm, combined with Monte Carlo sampling, is used to calculate and filter the weights of each wavelength variable in the spectrum to obtain highly correlated characteristic wavelengths. The union of the low collinearity characteristic wavelengths and the highly correlated characteristic wavelengths is taken to obtain the core spectral features. A CNN-partial least squares regression hybrid algorithm is used, with the selected core spectral features as input, to output the accurate measurement value of the sample's GRA content.

[0006] Furthermore, the enhanced RGB image is obtained through bilateral filtering and multi-scale MSR algorithms to extract the ROI image, including the following steps: Noise removal is achieved using a bilateral filtering algorithm. Starting from pixel spatial proximity, a spatial domain kernel is used to weight spatially adjacent pixels within the filtering window, with pixels farther away receiving lower weights, thus avoiding spatial diffusion issues caused by global filtering. Simultaneously, a pixel value domain kernel is used to smooth pixels with similar grayscale values ​​within the filtering window, based on pixel value similarity. Subsequently, a multi-scale MSR algorithm is employed to perform illumination correction on the denoised RGB image processed by the bilateral filtering algorithm. The denoised RGB image is transformed into a logarithmic domain image, converting the product relationship between illumination and reflection components in the denoised RGB image into a summation relationship in the logarithmic domain. Convolution operations are performed on the logarithmic domain image using Gaussian kernels of different scales to obtain illumination component estimates at different scales. Subtracting the illumination component estimates from the logarithmic domain image yields reflection component estimates at different scales. These reflection component estimates at different scales are then weighted and averaged to obtain the fused reflection component. Finally, the fused reflection component is converted from the logarithmic domain to the real domain and linearly stretched to map pixel values ​​to 0-255, resulting in an enhanced RGB image. The enhanced RGB image is processed and input into the YOLOv8 backbone network. This backbone network consists of four stages, each comprising a C2f module and convolutional layers. The C2f module uses depthwise separable convolution, including two steps: depthwise convolution and pointwise convolution. Depthwise convolution performs individual convolution on each channel to extract spatial features, while pointwise convolution uses... Convolutional processing is performed along the channel dimension, with stage 2 outputting shallow features, stage 3 outputting mid-level features, and stage 4 outputting deep features. Subsequently, the YOLOv8 neck network employs a feature pyramid network and path aggregation network structure, embedding a CBAM attention mechanism. The fusion is achieved through a channel attention module, which uses global average pooling and global max pooling to capture global contextual information along the channel dimension, adaptively strengthening the weights of effective feature channels related to the leaf target. A spatial attention module captures the positional information of the leaf target in the spatial dimension based on the output of the channel attention module, strengthening the feature weights of the leaf region. Simultaneously, the shallow, mid-level, and deep features output from the backbone network are fused to obtain a multi-scale fused feature map. This multi-scale fused feature map is input into the YOLOv8 detection head, which includes a detection branch and a segmentation branch. The detection branch predicts the bounding box and classification confidence of the broccoli leaf target, while the segmentation branch predicts the mask features of the leaf. Finally, the detection head outputs an ROI image containing only the functional leaves of the broccoli, and performs size normalization.

[0007] Furthermore, extracting multimodal features from the ROI image includes the following steps: Morphological coding features are extracted through a morphological coding feature extraction branch. The ROI image is converted into a single-channel grayscale image, and the Otsu method is used to binarize the grayscale image to obtain a binary mask image of the functional leaves of broccoli. Then, OpenCV is used to extract the geometric morphological features of the leaves. Based on the native RGB color space of the ROI image, the ROI image is converted into HSV and Lab color spaces. For each channel in the three color spaces, the mean, standard deviation, skewness, kurtosis, and entropy of all pixels in the channel are calculated. Then, the channels and color spaces are concatenated to obtain the color features. The ROI image is converted into a single-channel grayscale image. First, the GLCM features are calculated from four different directions using the gray-level co-occurrence matrix, and the average value of the GLCM features in the four directions is taken as the global texture. The process begins by removing irregular patterns with noise interference using a uniform mode LBP algorithm and calculating a normalized histogram to obtain local texture features. Global texture features are then concatenated with local texture features to obtain overall texture features. Geometric morphology features, color features, and texture features are concatenated to obtain initial morphological features. A linear initial screening and nonlinear fine screening strategy is then used to select the core features with the highest correlation to broccoli GRA content from the initial morphological coding features. The ReliefF algorithm is then used, assigning high weights to features that can distinguish different categories and low weights to features with no distinguishing ability. All features are sorted in descending order of weight, and the top-ranked features are selected as the core correlation features. Finally, a two-layer fully connected network maps the low-dimensional core correlation features to high-dimensional morphological coding features. Deep visual coding features are extracted through a deep visual coding feature extraction branch. The deep visual coding feature extraction branch uses FasterNet as the base network, which includes the FasterNet backbone network, the FasterNet neck network, and the FasterNet prediction head. In the FasterNet backbone network, a channel attention mechanism is introduced to capture the long-distance dependencies between channels to obtain deep visual feature maps. The FasterNet neck network also uses a feature pyramid network structure to process the deep visual feature maps. The FasterNet prediction head uses an average global pooling layer to compress the output of the FasterNet neck network and then inputs it into two fully connected layers to obtain deep visual coding features. Morphological coding features and deep visual coding features are subjected to layer normalization to obtain normalized morphological coding features and normalized deep visual coding features. The two normalized features are then concatenated by channel to obtain concatenated features, which are then input into a multi-head self-attention layer. A query matrix, key matrix, and value matrix are generated through linear transformation, and the outputs of each attention head are calculated. The outputs of all attention heads are concatenated to obtain multi-head fusion features, which are then subjected to layer normalization and nonlinear transformation using activation functions to obtain cross-modal interaction features. Subsequently, the cross-modal interaction features are decomposed along the channel dimension to obtain morphological weighted features and deep visual weighted features, which are then processed by channel-based methods. The weight vector is calculated by passing two fully connected layers and activation functions to the two weighted guided features. The normalized morphological coding features and normalized deep visual coding features are multiplied element-wise with the corresponding weight vector to complete the adaptive weighted enhancement of features, and then concatenated along the channel dimension to obtain the final multimodal features. The multimodal features output by the cross-modal attention fusion module are input into the GRA content binary classification screening model. Through a three-layer fully connected classification network, the probability that broccoli belongs to high GRA content is output. All broccoli with probabilities lower than the preset probability threshold are removed, and all broccoli functional leaves with probabilities greater than or equal to the preset probability threshold are used as samples to be refined.

[0008] Furthermore, the GRA content of the samples retained from the initial screening is measured to output accurate values, including the following steps: For the samples retained from the initial screening and awaiting further testing, the core features that can distinguish different categories of broccoli, calculated based on the ReliefF algorithm, were used. The five features with the highest weights were selected, and the regions corresponding to these five high-weight features were located in the ROI image. Morphological opening operations were then used to remove the main vein region, generating a binary mask for the morphologically highly correlated region. Subsequently, based on the deep visual feature map output by the FasterNet backbone network, the Grad-CAM mapping algorithm was used, with the result of the GRA content binary classification initial screening model as the target, to calculate the gradient of the deep visual feature map, obtaining a binary mask for the deep visually highly activated region. A pixel-by-pixel intersection operation was performed on the morphologically highly correlated region binary mask and the deep visually highly activated region binary mask to obtain an intersection region that simultaneously satisfies the requirements of both categories. Connectivity analysis was performed on the intersection region to obtain the connected components. Finally, a circular region with a preset diameter, centered on the centroid of the selected connected components, was designated as the target acquisition area for spectral acquisition, obtaining spectral data. The acquired spectral data is denoised using a smoothing filtering algorithm to obtain smoothed spectral data. Then, the smoothed spectral data is standardized using the Z-score normalization method to obtain standard spectral data. A continuous projection algorithm is used to filter out low-collinearity characteristic wavelengths, and a competitive adaptive reweighting algorithm is used to obtain high-correlation characteristic wavelengths. The union of the low-collinearity and high-correlation characteristic wavelengths is taken to obtain the core spectral features. Each wavelength in the standard spectral data is considered as its corresponding spectral vector. Specifically, in the continuous projection algorithm, a reference wavelength is selected and added to the low-collinearity set. For all other wavelengths, the projection of the corresponding spectral vector onto the spectral vector of the initial wavelength is calculated. For the shadow vector, calculate the magnitude of the projection vector corresponding to all wavelengths, add the wavelength with the largest magnitude of the projection vector to the low collinearity set, and use this wavelength as the reference wavelength for the next iteration; repeat the above iterative process to obtain the low collinearity feature wavelengths; in the competitive adaptive reweighting algorithm, randomly select 80% of the wavelengths from the standard spectral data, calculate the importance score of the corresponding spectral vector for each selected wavelength, sort the importance scores from largest to smallest, and retain the wavelength with the highest importance score to enter the high correlation set; repeat the above steps, and after reaching the required number of iterations, use the wavelengths in the high correlation set as the high correlation feature wavelengths; finally, take the union of the low collinearity feature wavelengths and the high correlation feature wavelengths to obtain the core spectral features; A hybrid CNN-partial least squares regression algorithm is employed, using the selected core spectral features as input to output precise measurements of GRA content in samples. Specifically, the CNN is responsible for capturing the nonlinear relationship between core spectral features and broccoli GRA content, extracting deep spectral coding features, while the partial least squares regression model is responsible for establishing a stable linear mapping relationship between deep spectral coding features and broccoli GRA content. The CNN structure consists of two stacked convolutional-pooling layers. The core spectral features are extracted through the convolutional-pooling layers to obtain a feature map, capturing higher-dimensional nonlinear features and uncovering subtle changes in the spectrum related to broccoli GRA content. Finally, a global average pooling layer is used to perform global average pooling on the feature map to obtain deep spectral coding features. Subsequently, the deep spectral coding features are input into the partial least squares regression model. Through principal component orthogonal transformation, the deep spectral coding features are projected onto uncorrelated principal component spaces. Using a fixed regression coefficient matrix, precise measurements of GRA content in the sample to be precisely measured are output.

[0009] Compared with existing technologies, this invention constructs a two-level hierarchical detection architecture of rapid and non-destructive initial screening and targeted precise quantification. It achieves rapid and non-destructive initial screening of large batches of samples through multimodal feature fusion of ordinary RGB images, improving the efficiency of large-batch sample detection. It constructs a cross-modal attention fusion mechanism of morphological features and deep visual features, establishing an effective mapping relationship between appearance visual features and internal GRA content. For samples retained in the initial screening for further testing, it combines targeted spectral acquisition with core feature wavelength selection. It locates the targeted acquisition area through visual features, improving the targeting and effectiveness of spectral acquisition. Feature wavelength selection reduces the dimensionality of spectral features, decreasing the computational load and time consumption of spectral detection. Finally, it combines a CNN-PLSR hybrid algorithm, balancing the nonlinear representation capability of spectral features with the stability of linear regression, improving detection accuracy. Attached Figure Description

[0010] Figure 1 The flowchart shows the GRA grading and detection method for broccoli based on multimodal fusion and spectral quantification. Figure 2 Flowchart of the method for extracting ROI images; Figure 3 Flowchart for extracting multimodal features from a ROI image; Figure 4 A flowchart for the accurate detection of GRA content. Detailed Implementation

[0011] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0012] like Figure 1 As shown, a specific embodiment of the present invention discloses a method for GRA grading and detection of broccoli based on multimodal fusion and spectral quantification, comprising the following steps: RGB images of functional leaves of broccoli were acquired. The RGB images of functional leaves of broccoli were preprocessed using bilateral filtering algorithm and multi-scale MSR algorithm to obtain enhanced RGB images. An attention mechanism was introduced into the YOLOv8 segmentation network to extract the region of interest (ROI) image from the enhanced RGB image. A dual-branch feature extraction architecture is constructed, consisting of morphological coding feature extraction and deep visual coding feature extraction. The morphological coding feature extraction branch extracts morphological coding features from the ROI image, and the deep visual coding feature extraction branch extracts deep visual coding features from the ROI image. A cross-modal attention fusion module achieves adaptive complementary fusion of the two types of features to obtain multimodal features, which are then input into a binary classification screening model for GRA content to initially screen broccoli and obtain samples for further testing. For the samples retained from the initial screening, based on the multimodal features extracted during the initial screening process, the representative region in the leaves with the highest correlation to GRA content is located as the target region for spectral acquisition. Spectral data is acquired from the target region. A continuous projection algorithm is used to project and transform the spectral data to obtain low collinearity characteristic wavelengths. A competitive adaptive reweighting algorithm, combined with Monte Carlo sampling, is used to calculate and filter the weights of each wavelength variable in the spectrum to obtain highly correlated characteristic wavelengths. The union of the low collinearity characteristic wavelengths and the highly correlated characteristic wavelengths is taken to obtain the core spectral features. A CNN-partial least squares regression hybrid algorithm is used, with the selected core spectral features as input, to output the accurate measurement value of the sample's GRA content.

[0013] Furthermore, RGB images of broccoli leaves are acquired according to a pre-set resolution and shooting distance, wherein the acquisition resolution is fixed. Furthermore, the distance between the broccoli and its functional leaves should be controlled at 30cm during collection, and the collection angle should be perpendicular to the plane of the leaf.

[0014] Furthermore, such as Figure 2 As shown, the enhanced RGB image is obtained by processing with bilateral filtering algorithm and multi-scale MSR algorithm to extract ROI image, including the following steps: To eliminate the interference of complex field environments on the collected RGB images of broccoli functional leaves and retain effective features related to GRA content, a bilateral filtering algorithm was first used for noise removal. Starting from pixel spatial proximity using a spatial domain kernel, pixels spatially adjacent within the filtering window were weighted, with pixels farther apart receiving lower weights to avoid spatial diffusion problems caused by global filtering. Simultaneously, starting from pixel value similarity using a pixel value domain kernel, pixels with similar grayscale values ​​within the filtering window were smoothed, while pixels with large grayscale differences retained their original grayscale values, thus protecting the edges of the broccoli functional leaves. Finally, a multi-scale MSR algorithm was used to refine the denoised RGB image processed by the bilateral filtering algorithm. Illumination correction is performed to eliminate uneven brightness and color distortion in images caused by complex field environments. The denoised RGB image is transformed into a logarithmic domain image by logarithmic domain transformation. The product relationship between the illumination component and the reflection component in the denoised RGB image is transformed into a summation relationship in the logarithmic domain. The logarithmic domain image is convolved with Gaussian kernels of different scales to obtain illumination component estimates at different scales. The illumination component estimates are subtracted from the logarithmic domain image to obtain reflection component estimates at different scales. The reflection component estimates at different scales are weighted and averaged to obtain the fused reflection component. Finally, the fused reflection component is transformed from the logarithmic domain to the real domain and the pixel values ​​are mapped to 0-255 by linear stretching to obtain the enhanced RGB image. The enhanced RGB image, processed by bilateral filtering and multi-scale MSR algorithms, is input into the YOLOv8 segmentation network to accurately separate the functional leaves of broccoli from the field background, obtaining the ROI image. Specifically, the enhanced RGB image is processed, and the size is reduced from... Modified to The data is then fed into the YOLOv8 backbone network, which consists of four stages. Each stage comprises a C2f module and convolutional layers. The C2f module uses depthwise separable convolution, including two steps: depthwise convolution and pointwise convolution. The depthwise convolution performs individual convolution on each channel to extract spatial features, while the pointwise convolution uses... Convolutional processing is performed along the channel dimension, with stage 2 outputting shallow features, stage 3 outputting mid-level features, and stage 4 outputting deep features. Subsequently, the YOLOv8 neck network, employing a feature pyramid network and path aggregation network structure, embeds a CBAM attention mechanism. The fusion is achieved through a channel attention module, which captures global contextual information along the channel dimension based on global average pooling and global max pooling, adaptively strengthening the weights of effective feature channels related to the leaf target. A spatial attention module captures the spatial location information of the leaf target based on the output of the channel attention module, strengthening the feature weights of the leaf region. Simultaneously, the shallow, mid-level, and deep features output from the backbone network are fused to obtain a multi-scale fused feature map. This multi-scale fused feature map is input into the YOLOv8 detection head, which includes a detection branch and a segmentation branch. The detection branch predicts the bounding box and classification confidence of the broccoli leaf target, while the segmentation branch predicts the mask features of the leaf. Finally, the detection head outputs an ROI image containing only the functional leaves of the broccoli, which is then normalized to a smaller size. .

[0015] Furthermore, such as Figure 3 As shown, extracting multimodal features from a ROI image includes the following steps: A dual-branch feature extraction architecture is constructed, extracting morphological encoded features through a morphological encoding feature extraction branch and deep visual encoding features through a deep visual encoding feature extraction branch. Specifically, morphological encoded features are extracted through the morphological encoding feature extraction branch. The ROI image is converted into a single-channel grayscale image, and the Otsu method is used to binarize the grayscale image to obtain a binary mask image of the functional leaves of broccoli. Then, OpenCV is used to extract the geometric morphological features of the leaves. Based on the native RGB color space of the ROI image, the ROI image is converted into HSV and Lab color spaces. For each channel in the three color spaces, the mean, standard deviation, skewness, kurtosis, and entropy of all pixels in the channel are calculated. Then, the channels and color spaces are concatenated to obtain the color features. The ROI image is converted into a single-channel grayscale image. First, the GLCM features are calculated from four different directions using the Gray-Level Co-occurrence Matrices (GLCM), and the average value of the GLCM features in the four directions is taken as the global texture. The process involves several steps: First, irregular patterns with noise interference are removed using uniform mode LBP, and a normalized histogram is calculated to obtain local texture features. Global texture features are then concatenated with local texture features to obtain overall texture features. Geometric morphology features, color features, and texture features are concatenated to obtain initial morphological features. A linear initial screening and nonlinear fine screening strategy is then used to select the core features with the highest correlation to broccoli GRA content from the initial morphological coding features. The ReliefF algorithm is then used in multiple iterations. In each iteration, a broccoli functional leaf sample is randomly selected, and its 10 nearest neighbors of the same category and 10 nearest neighbors of different categories are found. Features that can distinguish different categories are assigned high weights, while features without discrimination are assigned low weights. After iteration, all features are sorted in descending order of weight, and the top-ranked features are selected as the core correlation features. Finally, a two-layer fully connected network maps the low-dimensional core correlation features to high-dimensional morphological coding features. Deep visual coding features are extracted through a deep visual coding feature extraction branch. This branch uses FasterNet as its base network, which includes the FasterNet backbone, FasterNet neck network, and FasterNet prediction head. A channel attention mechanism is introduced into the FasterNet backbone to capture long-range dependencies between channels, resulting in deep visual feature maps. The FasterNet neck network also uses a feature pyramid network structure to process the deep visual feature maps. The FasterNet prediction head uses an average global pooling layer to compress the output of the FasterNet neck network before inputting it into two fully connected layers to obtain deep visual coding features. FasterNet is a mature deep learning network model, and its channel attention mechanism has been described in the YOLOv8 segmentation network, so it will not be repeated here. Morphological encoding features and deep visual encoding features are input into a cross-modal attention fusion module for adaptive complementary fusion to obtain multimodal features. Specifically, firstly, the morphological encoding features and deep visual encoding features are subjected to layer normalization to obtain normalized morphological encoding features and normalized deep visual encoding features. The means and variances of the two features are aligned to eliminate distribution differences between the two features. The two normalized features are then concatenated by channel to obtain concatenated features, which are then input into a multi-head self-attention layer. A query matrix, key matrix, and value matrix are generated through linear transformation, and the outputs of each attention head are calculated. The outputs of all attention heads are concatenated to obtain multi-head fused features, which are then subjected to layer normalization and nonlinear transformation by activation functions to obtain cross-modal interaction features. Subsequently, the cross-modal interaction features are decomposed along the channel dimension to obtain... The morphological weight-guided features and deep visual weight-guided features are processed by two fully connected layers and activation functions to calculate weight vectors. Normalized morphological and deep visual encoding features are then element-wise multiplied with their corresponding weight vectors to achieve adaptive weighted enhancement, and finally concatenated along the channel dimension to obtain the final multimodal features. The multimodal features output from the cross-modal attention fusion module are input into a binary classification screening model for GRA content. A three-layer fully connected classification network outputs the probability that broccoli has high GRA content. Broccoli with probabilities below a preset threshold are removed, while functional leaves with probabilities greater than or equal to the preset threshold are used as samples for further testing. The multi-head self-attention mechanism is a mature mechanism in deep learning and will not be elaborated upon here.

[0016] Furthermore, such as Figure 4 As shown, the precise determination value of GRA content in the samples retained from the initial screening for further testing is output through the following steps: For the samples retained from the initial screening and awaiting further testing, based on the multimodal features extracted during the initial screening process, the representative regions in the leaves with the highest correlation to GRA content were located as the target areas for spectral acquisition. Specifically, based on the core features that can distinguish different types of broccoli calculated using the ReliefF algorithm, the five features with the highest weights were selected, and the regions corresponding to the five high-weight features were located in the ROI image. The main vein region was then removed using morphological opening operations to generate a binary mask of the morphologically highly correlated region. Subsequently, based on the deep visual feature map output by the FasterNet backbone network, the Grad-CAM mapping algorithm was used to calculate the GRA content binary classification initial screening model results. The gradient of the deep visual feature map is used to generate a heatmap. The higher the value in the heatmap, the greater the contribution of the region to the result of the binary classification screening model of GRA content. An appropriate activation threshold is set, and regions in the heatmap that are greater than or equal to the activation threshold are marked as valid regions, resulting in a binary mask of the high activation region of deep vision. The binary mask of the morphologically highly correlated region and the binary mask of the high activation region of deep vision are subjected to pixel-by-pixel intersection operation to obtain the intersection region that simultaneously meets the requirements of both categories. Connectivity analysis is performed on the intersection region to remove scattered regions with too small an area, resulting in connected regions. Finally, a circular region with a preset diameter is marked as the target acquisition area with the centroid of the selected connected region as the center for spectral acquisition, resulting in spectral data. The collected spectral data were denoised using a smoothing filtering algorithm to obtain smoothed spectral data. Then, the smoothed spectral data was standardized using the Z-score normalization method to obtain standard spectral data. Considering the presence of redundant and interfering bands in the standard spectral data, a continuous projection algorithm was used to screen out low-collinearity characteristic wavelengths to identify core characteristic bands related to broccoli GRA content. A competitive adaptive reweighting algorithm was used to obtain high-correlation characteristic wavelengths. The union of the low-collinearity and high-correlation characteristic wavelengths was then used to obtain the core spectral features. Each wavelength in the standard spectral data was considered as its corresponding spectral vector. Specifically, in the continuous projection algorithm, a reference wavelength was selected and added to the low-collinearity set. For all other wavelengths, the following steps were taken: Calculate the projection vector of the corresponding spectral vector onto the spectral vector of the initial wavelength. Calculate the magnitude of the projection vectors corresponding to all wavelengths. Add the wavelength with the largest magnitude of the projection vector to the low collinearity set and use this wavelength as the reference wavelength for the next iteration. Repeat the above iterative process to obtain the low collinearity feature wavelengths. In the competitive adaptive reweighting algorithm, randomly select 80% of the wavelengths from the standard spectral data. For each selected wavelength, calculate the importance score of its corresponding spectral vector. Sort the importance scores from largest to smallest and retain the wavelength with the highest importance score to enter the high correlation set. Repeat the above steps until the required number of iterations is reached. Then, use the wavelengths in the high correlation set as the high correlation feature wavelengths. Finally, take the union of the low collinearity feature wavelengths and the high correlation feature wavelengths to obtain the core spectral features. A hybrid CNN-partial least squares regression algorithm is employed, using the selected core spectral features as input to output precise measurements of GRA content in samples. Specifically, the CNN is responsible for capturing the nonlinear relationship between core spectral features and broccoli GRA content, extracting deep spectral coding features, while the partial least squares regression model is responsible for establishing a stable linear mapping relationship between deep spectral coding features and broccoli GRA content. The CNN structure consists of two stacked convolutional-pooling layers. The core spectral features are extracted through the convolutional-pooling layers to obtain a feature map, capturing higher-dimensional nonlinear features and uncovering subtle changes in the spectrum related to broccoli GRA content. Finally, a global average pooling layer is used to perform global average pooling on the feature map to obtain deep spectral coding features. Subsequently, the deep spectral coding features are input into the partial least squares regression model. Through principal component orthogonal transformation, the deep spectral coding features are projected onto uncorrelated principal component spaces. Using a fixed regression coefficient matrix, precise measurements of GRA content in the sample to be precisely measured are output.

[0017] Preferably, in the bilateral filtering algorithm, the spatial domain kernel uses a Gaussian kernel with a standard deviation of 15, the pixel value domain kernel uses a Gaussian kernel with a standard deviation of 25, and the filtering window size is [missing information]. In the multi-scale MSR algorithm, the standard deviations of the three Gaussian kernels with different scales are set to 15, 80, and 250, respectively. In the backbone network of the YOLOv8 segmentation network, the convolutional kernels corresponding to the four stages are all... The depthwise separable convolutions used are all Depth convolution and Pointwise convolutions are used; in the neck network of the YOLOv8 segmentation network, a feature pyramid network and path aggregation network structure are adopted. In the embedded CBAM attention module, the channel attention module contains a fully connected layer, which reduces the number of input channels to one-sixteenth of the dimensionality. In the spatial attention module, a pooling layer is used after pooling. The convolutional kernels; the YOLOv8 segmentation network was trained using the AdamW optimizer with an initial learning rate of 0.001, a batch size of 16, a maximum number of training epochs of 100, and a confidence threshold of 0.5 in the detection head; Preferably, in the dual-branch feature extraction architecture, the geometric morphological features extracted by the morphological coding feature extraction branch are 22-dimensional, the color features are 45-dimensional, the global texture features are 4-dimensional, the local texture features are 59-dimensional, the initial morphological features are 130-dimensional, the associated core features are 30-dimensional, and the dimensions of the two fully connected layers are... The morphological encoding features are 256-dimensional; the input to the FasterNet backbone is 256-dimensional, containing 4 stages, with convolutional kernels of size [missing value]. The output of the FasterNet neck network is 80-dimensional. The dimensions of the two fully connected layers in the FasterNet head sampling prediction are... The output deep visual coding features are 256-dimensional. This dual-branch feature extraction architecture is trained using the AdamW optimizer with a batch size of 32 and an initial learning rate of 0.001, which decays at a rate of 0.5 every 10 rounds. Preferably, in the cross-modal attention fusion module, the number of attention heads in the multi-head self-attention layer is 8, each attention head has a dimension of 64, and the two fully connected layers used to generate the weight vector are... The final output multimodal feature dimension is 512; in the GRA content binary classification screening model, the dimension of the three fully connected layer classification network is... The Adam optimizer was used for training, with a weighted cross-entropy loss function, a batch size of 64, a cosine annealing strategy, an initial learning rate of 0.001, and a decay rate of 0.5 every 10 rounds. In addition, the preset probability threshold for binary classification screening was set to 0.75. Preferably, RGB images of functional leaves of broccoli and the corresponding GRA content of broccoli are collected as training data, totaling 1500 samples. These samples are divided into training, testing, and validation sets in a 7:2:1 ratio. Among the core features with the highest correlation to broccoli GRA content selected from the initial morphological coding features, the Pearson correlation coefficient is used to calculate the Pearson correlation coefficient between each feature in the initial morphological coding features and the corresponding sample's broccoli GRA content. Features with an absolute Pearson correlation coefficient greater than or equal to 0.3 are retained, while features with an absolute Pearson correlation coefficient less than 0.3 are removed. In the competitive adaptive reweighting algorithm, the importance score is calculated by calculating the Pearson correlation coefficient between the spectral vector and the corresponding sample's broccoli GRA content, and the absolute value of the Pearson correlation coefficient corresponding to each wavelength is used as its importance score. The heatmap activation threshold is set to 0.7, the preset diameter of the target region is set to 0.9 cm, the maximum number of iterations for the continuous projection algorithm is 30, and the maximum number of iterations for the competitive adaptive weighted algorithm is 30. In the CNN-partial least squares regression hybrid algorithm, the CNN structure consists of two stacked convolutional-pooling layers. The first convolutional-pooling layer contains 32 convolutional kernels with a window size of 2 and a stride of 2. The second convolutional-pooling layer contains 64 convolutional kernels with a window size of 2 and a stride of 2. The deep spectral encoding features are 64-dimensional. In the partial least squares regression model, the number of principal components is set to 8. A linear regression mapping between the principal components and the broccoli GRA content is used to obtain the regression coefficient matrix. The AdamW optimizer is used for training, with a batch size of 32 and an initial learning rate of 0.001. The learning rate decays at a rate of 0.5 every 10 rounds. The cross-entropy between the precise measured value of the GRA content of the sample to be precisely measured and the corresponding broccoli GRA content is calculated as the loss function. Training ends when the loss function does not continue to decrease after 3 consecutive rounds of training.

[0018] This invention discloses a grading detection method for GRA in broccoli based on multimodal fusion and spectral quantification. It constructs a two-level grading detection architecture of rapid, non-destructive initial screening followed by targeted, precise quantification. Rapid, non-destructive initial screening of large batches of samples is achieved through multimodal feature fusion of ordinary RGB images, improving the efficiency of large-batch sample detection. A cross-modal attention fusion mechanism combining morphological features and deep visual features is constructed to establish an effective mapping relationship between appearance visual features and internal GRA content. For samples retained from the initial screening, targeted spectral acquisition is combined with core feature wavelength selection. Targeted acquisition areas are located through visual features, improving the targeting and effectiveness of spectral acquisition. Feature wavelength selection reduces the dimensionality of spectral features, decreasing the computational load and time consumption of spectral detection. Finally, a CNN-PLSR hybrid algorithm is combined to balance the nonlinear representation capability of spectral features with the stability of linear regression, improving detection accuracy.

[0019] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for GRA grading and detection of broccoli based on multimodal fusion and spectral quantification, characterized in that, Includes the following steps: RGB images of functional leaves of broccoli were acquired, and the RGB images of functional leaves of broccoli were preprocessed to obtain enhanced RGB images. The CBAM attention mechanism was introduced into the YOLOv8 segmentation network to extract the region of interest (ROI) images. A dual-branch feature extraction architecture is constructed. Morphological coding features are extracted from ROI images through a morphological coding feature extraction branch, and deep visual coding features are extracted through a deep visual coding feature extraction branch. Multimodal features are obtained by adaptive complementary fusion through a cross-modal attention fusion module and input into a binary classification screening model for GRA content to obtain samples to be refined. Based on the multimodal features extracted during the initial screening process, the target region with the highest correlation to GRA content in the leaves was located, and spectral data was obtained. A continuous projection algorithm was used for projection transformation to obtain low collinearity feature wavelengths. A competitive adaptive reweighting algorithm was used to filter the weights of each wavelength variable in the spectrum to obtain highly correlated feature wavelengths. The union of the low collinearity feature wavelengths and the highly correlated feature wavelengths was taken to obtain the core spectral features. A CNN-partial least squares regression hybrid algorithm was used, with the core spectral features obtained as input, to output the accurate measurement value of GRA content in the sample.

2. The method for GRA grading and detection of broccoli based on multimodal fusion and spectral quantification as described in claim 1, characterized in that, The method for calculating the multimodal features includes the following steps: Construct a parallel dual-branch architecture for morphological coding feature extraction and deep visual coding feature extraction; Geometric morphological features, color features, and texture features are extracted from the ROI image through a morphological coding feature extraction branch. The three features are then concatenated to obtain the initial morphological features. Linear initial screening and nonlinear fine screening are used to perform multiple rounds of screening on the initial morphological features to obtain associated core features. A high-dimensional mapping operation is then performed on the associated core features through a fully connected layer network to obtain the morphological coding features. Deep visual feature maps are extracted from ROI images through a deep visual coding feature extraction branch. Deep visual coding features are obtained through multi-scale fusion and dimensionality compression operations. Layer normalization is performed on morphological encoding features and deep visual encoding features to obtain two types of normalized features. The two types of normalized features are concatenated to obtain concatenated features, which are then input into a multi-head self-attention layer to obtain cross-modal interaction features. The cross-modal interaction features are decomposed to obtain two types of weighted guided features, which are then passed through a fully connected layer to obtain the corresponding weight vectors. The corresponding normalized features are then weighted element-wise using the weight vectors to obtain two types of weighted enhanced features, which are then concatenated to obtain the final multimodal features.

3. The method for GRA grading and detection of broccoli based on multimodal fusion and spectral quantification as described in claim 2, characterized in that, The method for obtaining the target region includes the following steps: Based on the core features of association, the five features with the highest weights are selected. Regions corresponding to these five high-weight features are located in the ROI image, and the main vein region is removed using morphological opening operations to generate a binary mask of morphologically highly correlated regions. Subsequently, based on the deep visual feature map, the Grad-CAM mapping algorithm is used, with the result of the GRA content binary classification screening model as the target, to calculate the gradient of the deep visual feature map, generating a heatmap. An appropriate activation threshold is set, and regions in the heatmap greater than or equal to the activation threshold are marked as valid regions, resulting in a binary mask of deep visually highly activated regions. A pixel-by-pixel intersection operation is performed on the morphologically highly correlated region binary mask and the deep visually highly activated region binary mask to obtain an intersection region that simultaneously meets the requirements of both classes. Connectivity analysis is performed on the intersection region, and scattered regions with excessively small areas are removed to obtain connected regions. Finally, a circular region with a preset diameter is marked as the target region, centered on the centroid of the selected connected region.

4. The method for GRA grading and detection of broccoli based on multimodal fusion and spectral quantification as described in claim 1, characterized in that, The method for calculating the core spectral features includes the following steps: The collected spectral data is denoised and standardized to obtain standard spectral data; In the continuous projection algorithm, a reference wavelength is selected from the standard spectral data and added to the low collinearity set. For all other wavelengths, the projection vector of the corresponding spectral vector onto the spectral vector of the initial wavelength is calculated. The magnitude of the projection vectors corresponding to all wavelengths is calculated. The wavelength with the largest magnitude of the projection vector is added to the low collinearity set and used as the reference wavelength for the next iteration. The above iterative process is repeated to obtain the low collinearity characteristic wavelengths. In the competitive adaptive reweighting algorithm, wavelengths are randomly selected from standard spectral data. For each selected wavelength, the importance score of its corresponding spectral vector is calculated. The importance scores are sorted from largest to smallest, and the wavelengths with the highest importance scores are retained and added to the high-relevance set. The above steps are repeated until the number of iterations is reached, and the wavelengths in the high-relevance set are used as high-relevance feature wavelengths. The core spectral features are obtained by taking the union of the low collinearity characteristic wavelengths and the high correlation characteristic wavelengths.

5. The method for GRA grading and detection of broccoli based on multimodal fusion and spectral quantification as described in claim 1, characterized in that, The method for calculating the accurate GRA content of the sample includes the following steps: A hybrid regression algorithm architecture combining convolutional neural networks (CNNs) and partial least squares regression models is constructed. Core spectral features are input into the CNN, and convolutional operations are performed on the core spectral features through the convolutional layers of the CNN to extract local correlation features. The local correlation features are downsampled through the pooling layers of the CNN. The convolutional operation and downsampling are repeated to obtain a high-dimensional feature map. Global average pooling is performed on the high-dimensional feature map to obtain deep spectral coding features. The deep spectral coding features are input into the partial least squares regression model, and principal component orthogonal transformation is performed. The transformed features are then projected onto the uncorrelated principal component space to obtain principal component features. Based on the principal component features and the GRA content standard, a linear regression mapping operation is performed to obtain a fixed regression coefficient matrix. The input deep spectral coding features are then processed through the regression coefficient matrix to output the accurate measurement value of the GRA content of the sample to be precisely measured.

6. The method for GRA grading and detection of broccoli based on multimodal fusion and spectral quantification as described in claim 1, characterized in that, The method for obtaining the ROI image includes the following steps: The enhanced RGB image is input into the YOLOv8 segmentation network. Each channel is convolved individually through the YOLOv8 backbone network to extract spatial features. Pointwise convolution is then used to fuse the features along the channel dimension, outputting deep features. Subsequently, the YOLOv8 neck network, employing a feature pyramid network and path aggregation network structure with embedded CBAM attention mechanism, is used to fuse the features through the channel attention module, resulting in a multi-scale fused feature map. The multi-scale fused feature map is then input into the YOLOv8 detection head, outputting an ROI image containing the functional leaves of broccoli.

7. The method for GRA grading and detection of broccoli based on multimodal fusion and spectral quantification as described in claim 2, characterized in that, The method for calculating the associated core features includes the following steps: Linear correlation calculation is performed on each feature dimension in the initial morphological features to obtain the correlation coefficient between each feature dimension and the GRA content of the corresponding sample, and the absolute value is taken; a correlation coefficient screening threshold is set, and features with absolute values ​​greater than or equal to the screening threshold are retained to complete the linear initial screening operation; The ReliefF algorithm is used to perform nonlinear weight calculation on the features after linear initial screening. In each iteration, a single functional leaf sample of broccoli is randomly selected, and the selected sample is matched with similar and dissimilar neighbor samples. Based on the feature differences between similar and dissimilar neighbor samples, a weight is assigned to each feature dimension. All feature dimensions are sorted in descending order of weight, and the feature dimensions with the highest ranking are selected to obtain the core features with the highest correlation to the GRA content of broccoli.

8. The method for GRA grading and detection of broccoli based on multimodal fusion and spectral quantification as described in claim 1, characterized in that, The method for generating the enhanced RGB image includes the following steps: Noise removal is achieved using a bilateral filtering algorithm. A spatial domain kernel is used to weight spatially adjacent pixels within the filtering window based on pixel spatial proximity. Simultaneously, a pixel value domain kernel is used to smooth pixels with similar grayscale values ​​within the filtering window based on pixel value similarity, resulting in a denoised RGB image. Subsequently, a multi-scale MSR algorithm is used to transform the denoised RGB image into a logarithmic domain image, converting the product relationship between the illumination and reflection components in the denoised RGB image into a summation relationship in the logarithmic domain. Convolution operations are performed on the logarithmic domain image using Gaussian kernels of different scales to obtain illumination component estimates at different scales. Subtracting the illumination component estimates from the logarithmic domain image yields reflection component estimates at different scales. These reflection component estimates at different scales are then weighted and averaged to obtain a fused reflection component. Finally, the fused reflection component is converted from the logarithmic domain to the real domain and linearly stretched to map pixel values ​​to 0-255, resulting in an enhanced RGB image.

9. The method for GRA grading and detection of broccoli based on multimodal fusion and spectral quantification as described in claim 1, characterized in that, The method for calculating the standard spectral data includes the following steps: The acquired spectral data is denoised using a smoothing filtering algorithm to obtain smoothed spectral data. Then, the smoothed spectral data is standardized using the Z-score normalization method to obtain standard spectral data.

10. The method for GRA grading and detection of broccoli based on multimodal fusion and spectral quantification as described in claim 2, characterized in that, The method for calculating the texture features includes the following steps: The ROI image is converted into a single-channel grayscale image. First, GLCM features are calculated from four different directions using the gray-level co-occurrence matrix, and the average value of the GLCM features in the four directions is taken as the global texture feature. Irregular patterns with noise interference are removed by uniform mode LBP, and a normalized histogram is calculated on the result to obtain the local texture features. The global texture features and the local texture features are then concatenated to obtain the texture features.

Images