Method for detecting content of soluble solids in strawberries based on multi-modal fusion

Abstract

The invention is suitable for the technical field of agricultural detection, and provides a strawberry soluble solid content detection method based on multi-modal fusion, and the method comprises the following steps: synchronously collecting a hyperspectral image, an RGB image and colorimetric parameters of a strawberry sample; preprocessing the hyperspectral image, and screening out a sensitive spectral interval related to SSC from a full spectrum based on MDWPLS; splicing the sensitive spectrum interval data with the colorimetric parameters to form a one-dimensional fusion feature vector, and converting the one-dimensional fusion feature vector into a two-dimensional GASF image through GASF; constructing a three-branch MTGBiT-MHA fusion model, and performing feature extraction on the one-dimensional fusion feature vector, the GASF image and the RGB image; and performing dynamic weighted fusion on the features extracted by the three-branch MTGBiT-MHA fusion model by using MHA, and outputting a strawberry SSC predicted value. According to the method, high-precision and high-robustness nondestructive testing of the SSC of the strawberries with different maturity degrees can be realized.

Description

Technical Field

[0001] This invention belongs to the field of agricultural testing technology, and in particular relates to a method for detecting the soluble solids content of strawberries based on multimodal fusion. Background Technology

[0002] The soluble solids content (SSC) of strawberries is a key intrinsic quality indicator that determines their taste, flavor, and market value. Traditional testing methods rely on destructive sampling and refractometer measurements, which are inefficient and cannot achieve non-destructive testing of individual fruits, making them unsuitable for the needs of modern precision agriculture and post-harvest intelligent sorting.

[0003] Existing nondestructive testing technologies are mainly divided into three categories: (1) Spectrum-based techniques: such as near-infrared spectroscopy (NIRS) and hyperspectral imaging (HSI). They can reflect the chemical composition information inside the fruit, but they are sensitive to the surface condition of the fruit (humidity, texture), and the high-dimensional data is redundant and computationally complex.

[0004] (2) Machine vision-based technologies: such as RGB imaging. It can quickly acquire external phenotypic features such as color and shape, and is low in cost, but it cannot directly quantify internal sugar content and is easily affected by ambient light.

[0005] (3) Colorimetric techniques: such as CIE LAB colorimetric parameters. It can objectively quantify color changes and is related to maturity, but it lacks direct molecular-level information and has limited ability to distinguish subtle changes in SSC within the same maturity level.

[0006] Existing technologies attempting to fuse multimodal information typically employ simple feature vector concatenation. However, one-dimensional spectral data and two-dimensional image data differ fundamentally in structure (heterogeneity), and simple concatenation cannot deeply explore the nonlinear complementary relationships between modes. Furthermore, traditional spectral feature selection methods (such as BIPLS and IVISSA) are insufficient in balancing synergistic effects and noise resistance, making it difficult to accurately select stable feature bands strongly correlated with SSC from high-dimensional data. Therefore, a systematic technical solution capable of deeply fusing heterogeneous multi-source data and possessing efficient feature selection capabilities is urgently needed to achieve stable and high-precision non-destructive detection of SSC in strawberries at different maturity levels. Summary of the Invention

[0007] The purpose of this invention is to provide a method for detecting the soluble solids content of strawberries based on multimodal fusion, aiming to solve the problems mentioned in the background art.

[0008] The present invention is implemented as follows: a method for detecting the soluble solids content of strawberries based on multimodal fusion includes the following steps: Step 1: Multi-source data acquisition; Simultaneously acquire hyperspectral images, RGB images, and colorimetric parameters of strawberry samples; Step 2: Spectral feature screening; The hyperspectral image is preprocessed, and sensitive spectral regions related to SSC are screened from the full spectrum based on MDWPLS. Step 3: Multimodal data conversion; The sensitive spectral range data selected in step 2 are concatenated with the colorimetric parameters to form a one-dimensional fusion feature vector, and then the one-dimensional fusion feature vector is converted into a two-dimensional GASF image through GASF. Step 4: Constructing the three-branch model; Construct a three-branch MTGBiT-MHA fusion model, the model including: One-dimensional numerical branching uses a gated loop unit to process the one-dimensional fused feature vector composed of the sensitive spectral data and colorimetric parameters; Two-dimensional implicit branching is performed using BiT processing on the GASF image obtained in step 3; Two-dimensional explicit branching, using BiT to process RGB images; Step 5: Multi-head attention mechanism (MHA) fusion and output; The features extracted from one-dimensional numerical branches, two-dimensional implicit branches, and two-dimensional explicit branches are dynamically weighted and fused using MHA, and the predicted value of strawberry SSC is output through BiLSTM and a fully connected layer.

[0009] In a further technical solution, in step 2, the preprocessing of the hyperspectral image is achieved through a synergistic combination of four algorithms: continuous wavelet transform (CWT), multiplicative scattering correction (MSC), Savitzky-Golay (SG) convolutional smoothing, and minimum-maximum normalization (MMN). CWT is used to correct baseline drift; MSC is used to effectively suppress scattering effects and reduce interference caused by physical sample fluctuations; SG convolution smoothing is used to reduce spectral noise and improve the signal-to-noise ratio; MMN is used to ensure the consistency of data numerical range.

[0010] In a further technical solution, step 2 of the MDWPLS method includes the following sub-steps: Step 2.1: Construct a full-spectrum partial least squares model and calculate the VIP value and regression coefficient RC at each wavelength point; Step 2.2: Divide the full spectrum into A continuous subinterval; Step 2.3: Calculate and assign dynamic weights to each sub-interval based on the VIP value and the absolute value of RC at the wavelength points within each sub-interval; Step 2.4: Construct a weighted PLS model based on the assigned weights, and through iterative optimization, eliminate redundant intervals according to the changes in model prediction performance to lock in the key spectral interval combination that contributes the most to SSC.

[0011] In a further technical solution, step 3 of the GASF conversion includes the following steps: Step 3.1: Normalize the one-dimensional fused feature vector; Step 3.2: Map the normalized sequence to a polar coordinate system, where the numerical values ​​are encoded as angles and the sequence indexes are encoded as radii; Step 3.3: Calculate the sum of angles between any two points in the sequence using GASF, construct the Gram matrix, and generate a two-dimensional GASF image.

[0012] The present invention provides a method for detecting the soluble solids content of strawberries based on multimodal fusion, the beneficial effects of which are as follows: (1) High detection accuracy: By accurately screening features through MDWPLS and deeply mining associations through GASF, and combining the complementary fusion of three-branch models, this invention has achieved excellent performance on the test set, which is significantly better than single-modal or traditional dual-modal fusion methods.

[0013] (2) High efficiency in feature utilization: The MDWPLS algorithm effectively compresses high-dimensional spectral data, greatly reducing data redundancy and computational load.

[0014] (3) Strong robustness and generalization ability: The model showed stable prediction performance on strawberry samples of three different maturity levels: mature, semi-mature and immature, effectively overcoming the interference caused by differences in fruit surface and changes in maturity.

[0015] (4) Enhanced interpretability: The fusion model organically combines internal chemical components (spectrum) with external phenotypic features (color, image texture), so that the prediction results have clear physiological significance. Attached Figure Description

[0016] Figure 1 A schematic diagram of a method for detecting soluble solids content in strawberries based on multimodal fusion, provided in an embodiment of the present invention; Figure 2 A schematic diagram of the structure of P in a method for detecting soluble solids content in strawberries based on multimodal fusion provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of P in a method for detecting the soluble solids content of strawberries based on multimodal fusion, provided in an embodiment of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0018] The specific implementation of the present invention will be described in detail below with reference to specific embodiments.

[0019] like Figure 1 and Figure 2 As shown, an embodiment of the present invention provides a method for detecting the soluble solids content of strawberries based on multimodal fusion, comprising the following steps: Step 1: Multi-source data acquisition and preprocessing; Hyperspectral and RGB images of strawberry samples were collected, along with colorimetric parameters (lightness L*, red-green axis a*, and yellow-blue axis b*). The hyperspectral images underwent black-and-white plate correction, and the chlorophyll index was used to determine the optimal parameters. Extracting the region of interest The image was preprocessed with a spectrum, and the RGB image was segmented to extract the pulp area and standardize its size.

[0020] Hyperspectral image preprocessing is achieved through a synergistic combination of four algorithms: CWT, MSC, SG convolutional smoothing, and MMN. CWT efficiently corrects baseline drift, while MSC effectively suppresses scattering effects and reduces interference caused by physical sample fluctuations. SG convolutional smoothing significantly improves the signal-to-noise ratio by reducing spectral noise. MMN ensures the consistency of data numerical ranges, facilitates the identification of overlapping spectral features, and maintains a constant dataset scale.

[0021] Step 2: Spectral feature selection based on the MDWPLS algorithm; Based on the partial least squares regression framework, this method divides the entire spectrum into intervals and assigns differentiated dynamic weights according to VIP values ​​and regression coefficients. An iterative optimization mechanism is then used to efficiently suppress insensitive background noise and information redundancy, while accurately identifying and retaining specific biochemical fingerprint bands with high interpretability for the detection target (i.e., SSC). The specific steps are as follows: Step 2.1: Construction of the full-spectrum partial least squares (PLS) basic model; First, an initial PLS model is constructed using preprocessed full-band spectral data, and a spectral matrix is ​​established. With the target vector of soluble solids content The mapping relationship between them; Dataset partitioning: The sample set is randomly divided into a calibration set (training set) and a validation set (test set) in a 7:3 ratio.

[0022] Performance benchmark establishment: Root mean square error of the full-spectrum model was calculated through cross-validation. and coefficient of determination This serves as a benchmark for evaluating the effectiveness of subsequent screening.

[0023] Step 2.2: Calculation of multidimensional feature contribution index; Based on the full-spectrum PLS model, the input bands are mapped to a low-dimensional latent variable space, and multidimensional statistical indicators reflecting the importance of each band are calculated, including: Regression coefficient RC: Characterizes the direct contribution rate of each wavelength point to the SSC prediction model.

[0024] Importance of projected variables: A comprehensive measure of the explanatory power of each wavelength variable on the model ( (Explanatory power) and predictive power ( The contribution of (correlation) to.

[0025] Loading Matrix: Analyzes the correlation between principal components and the original bands.

[0026] Step 2.3: Dynamic interval division and differentiated weight allocation; This is a core innovative step in MDWPLS, designed to quantitatively prioritize different spectral ranges: Interval division: Dividing the full spectrum into intervals A series of consecutive sub-intervals.

[0027] Weight Calculation: Based on the feature contribution indicators (mainly the absolute values ​​of VIP and RC) calculated in step 2, a weight coefficient is calculated for each sub-interval. Intervals containing bands with high VIP or high absolute RC values ​​will be assigned higher weights; conversely, intervals containing a large amount of noise or uninformative variables will be assigned lower weights. This process mathematically strengthens the definition of "sensitive intervals."

[0028] Step 2.4: Iterative weighted optimization and key interval locking; The optimal combination of intervals is selected through an iterative process, avoiding the instability of single-interval selection. Weighted modeling: Reconstructing a weighted PLS model using interval data with assigned weights.

[0029] Performance evaluation and elimination: Comparing the root mean square error of model predictions under different interval combinations and coefficient of determination ( If adding a certain interval leads to a reduction in model error () decline If the error increases (e.g., a rise), it is considered a critical interval and retained. If the addition of an interval leads to an increase in model error or no significant change, it is considered a redundant or interfering interval and removed.

[0030] Final output: After multiple rounds of iterative optimization, a set of discontinuous but highly correlated specific spectral bands is output as the final feature set. The feature interval combinations that contribute the most to SSC and have the least redundancy are retained (e.g., 620–640 nm, 720–760 nm, 920–960 nm).

[0031] Step 3: Multimodal data conversion based on GASF; This step aims to overcome the limitations of one-dimensional data in deep learning feature extraction by transforming physically heterogeneous data into two-dimensional images containing high-dimensional correlation information through Gram Angular Field (GASF); specifically: Step 3.1: Reconstruction of heterogeneous feature vectors; The specific spectral bands selected in step 2 are sequentially spliced ​​with the colorimetric parameters (L*, a*, b*) to construct a one-dimensional fusion feature vector containing "internal-external" quality information.

[0032] Step 3.2: Polar coordinate system mapping and encoding; First, the one-dimensional fused feature vector is normalized in the interval [-1, 1]. Then, the numerical sequence in the Cartesian coordinate system is mapped to the polar coordinate system using the inverse cosine function. In this process, the numerical magnitude is encoded as an angle, and the time / band index is encoded as a radius, thus introducing angular geometric features while preserving the original numerical distribution.

[0033] Step 3.3: GASF-related imaging; The Gramian matrix is ​​constructed by calculating the angle summation between any two points in the sequence using the Gram angle field summation method. This operation directly transforms the temporal / spectral correlations between elements in the one-dimensional fused feature vector into a two-dimensional spatial texture. The generated GASF image not only unifies the data dimensions of spectral data and RGB images (adapting to subsequent convolutional or Transformer networks), but more importantly, it visualizes the nonlinear synergistic effect between spectral bands and color parameters as a unique visual texture pattern, enabling the model to capture deep features that traditional regression methods cannot recognize through image recognition technology.

[0034] Step 4: Construct a three-branch MTGBiT-MHA fusion model; A three-branch deep learning architecture for heterogeneous data is designed, which decouples and extracts features for one-dimensional numerical data, two-dimensional implicit structures, and two-dimensional explicit phenotypes respectively (e.g., Figure 3 (as shown) One-dimensional numerical branch of GRU (time-dependent modeling): For the spectral bands and colorimetric parameter sequences after MDWPLS screening, a gated recurrent unit (GRU) is adopted. By utilizing its internal update gate and reset gate mechanism, the problem of gradient vanishing in long sequences is solved, the nonlinear mapping relationship in the "spectral fingerprint-physicochemical parameter" sequence is deeply explored, and the direct correlation between "component value-appearance color difference" is captured, providing direct quantitative numerical basis for SSC prediction.

[0035] GASF-BiT 2D Implicit Branch (Topology Mining): For 2D images generated by GASF transform, a bidirectional Transformer (BiT) is employed. GASF maps the numerical correlations of one-dimensional data to 2D spatial texture, and BiT utilizes its global self-attention mechanism to capture long-range pixel dependencies in the image. This branch aims to extract implicit, non-Euclidean co-topological features between spectral and color parameters (extracting implicit spectral-color space structural features, i.e., the potential patterns of "map-spectrum integration").

[0036] RGB-BiT 2D Explicit Branch (Phenographic Feature Encoding): For macroscopic strawberry fruit RGB images, the BiT architecture is also used. This branch focuses on extracting explicit visual phenotypic features of the fruit surface (such as color gradient distribution, texture defects, and maturity morphology), and utilizes the positional encoding characteristics of the Transformer to enhance the model's robustness to changes in fruit spatial pose and illumination.

[0037] Step 5: MHA Fusion and Output; In the feature fusion stage, instead of the traditional simple splicing, a deep fusion mechanism of "dynamic weighting-global awareness" was designed (e.g., Figure 3 (as shown) Dynamically weighted MHA: Introducing MHA into the fusion layer projects heterogeneous features onto multiple subspaces and computes attention scores in parallel. This mechanism enables adaptive feature alignment between modalities, i.e., dynamically assigning weights (soft selection) based on the contribution of features to the SSC, automatically enhancing key feature channels (such as the response of specific sensitive bands), while suppressing noise interference from redundant modes.

[0038] BiLSTM Global Context Awareness and Output: The weighted and fused feature sequence is input into a Bidirectional Long Short-Term Memory (BiLSTM) network. BiLSTM performs bidirectional temporal modeling within the fused feature space, further integrating global contextual information and preventing information loss during feature fusion. Finally, the deeply refined feature vectors are mapped through a fully connected layer (FC) to output high-precision SSC prediction values.

[0039] To validate this method, the "Hongyan" variety of strawberry was selected, and images in the 400-1000nm band were acquired using a Specim IQ hyperspectral camera; the L*, a*, and b* values ​​were measured using a CR-10 Plus colorimeter; and RGB images of the ROI region were simultaneously captured.

[0040] Feature Filtering (MDWPLS): The preprocessed full-band spectrum is input into the MDWPLS algorithm. The algorithm automatically identifies and outputs three core sensitive regions: 620–640 nm (corresponding to red light absorption), 720–760 nm (corresponding to the red edge region), and 920–960 nm (corresponding to moisture and sugar absorption). This step removes a large number of uninformative background bands.

[0041] Data Transformation (GASF): The reflectance data from the three intervals mentioned above are combined with the L*, a*, and b* values ​​to form a new feature vector. Applying the GASF algorithm, this vector is mapped into a two-dimensional heatmap (GASF image), where the texture pattern reflects the angular relationship between spectral values ​​and color parameters.

[0042] Model prediction (MTGBiT-MHA): Input: Input the filtered one-dimensional data into the GRU branch of the model; input the GASF image into the GASF-BiT branch; input the original RGB image of the pulp into the RGB-BiT branch.

[0043] Processing: The model automatically assigns weights through a multi-head attention mechanism and discovers the synergistic effect between spectral and colorimetric features (e.g., the a* value compensates for the insufficient sensitivity of the spectrum in the early maturation stage).

[0044] Output: The model outputs the predicted SSC value of this strawberry sample as 8.5°Brix.

[0045] Verification Results: Compared with the traditional destructive refractometer measurement (8.4°Brix), the error is only 0.1°Brix. In a test set containing 60 samples, the predicted values ​​of this embodiment method are in high agreement with the true values, demonstrating its effectiveness in practical applications.

[0046] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting soluble solids content of strawberry based on multi-modal fusion, characterized in that, The method comprises the following steps: Step 1: Multi-source data acquisition; Synchronously acquiring hyperspectral images, RGB images and colorimetric parameters of strawberry samples; Step 2: Spectral feature screening; Pretreating the hyperspectral images, and screening sensitive spectral intervals related to SSC from full spectrum based on MDWPLS; Step 3: Multi-modal data conversion; Splicing the sensitive spectral interval data screened in step 2 and the colorimetric parameters to form a one-dimensional fusion feature vector, and converting the one-dimensional fusion feature vector into a two-dimensional image through GASF to obtain a GASF image; Step 4: Three-branch model construction; Constructing a three-branch MTGBiT-MHA fusion model, and extracting features from the one-dimensional fusion feature vector, the GASF image and the RGB image respectively; Step 5: MHA fusion and output; Performing dynamic weighted fusion on the features extracted by the three-branch MTGBiT-MHA fusion model through MHA, and outputting the predicted value of strawberry SSC through BiLSTM and a full connection layer.

2. The method for detecting soluble solids content of strawberry based on multi-modal fusion according to claim 1, characterized in that, In the step 2, the pretreatment of the hyperspectral images is realized through the cooperative combination of four algorithms: CWT, MSC, SG convolution smoothing and MMN; CWT is used to correct baseline drift; MSC is used to suppress scattering effect and reduce interference caused by physical sample fluctuation; SG convolution smoothing is used to reduce spectral noise and improve signal-to-noise ratio; and MMN is used to ensure the consistency of the data value range.

3. The method for detecting soluble solids content of strawberry based on multi-modal fusion according to claim 2, characterized in that, In the step 2, the MDWPLS method comprises the following sub-steps: Step 2.1: Constructing a full-spectrum partial least squares model, and calculating the VIP value and the regression coefficient RC of each wavelength point; Step 2.2: Divide the full spectrum into individual consecutive sub-intervals; Step 2.3: According to the VIP value and the absolute value of RC of the wavelength points in each sub-interval, calculating and assigning a dynamic weight to each sub-interval; Step 2.4: Based on the assigned weight, constructing a weighted PLS model, and through iterative optimization, removing redundant intervals according to the change of model prediction performance, and locking the key spectral interval combination with the largest contribution to SSC.

4. The method for detecting soluble solids content of strawberry based on multi-modal fusion according to claim 3, characterized in that, In the step 3, the GASF conversion comprises the following steps: Step 3.1: Normalizing the one-dimensional fusion feature vector; Step 3.2: Mapping the normalized sequence to a polar coordinate system, in which the value is encoded as an angle and the sequence index is encoded as a radius; Step 3.3: Calculating the angle between any two points in the sequence through GASF, constructing a Gram matrix, and generating a two-dimensional GASF image.

5. The method for detecting soluble solids content of strawberry based on multi-modal fusion according to claim 4, characterized in that, In the step 4, the three-branch MTGBiT-MHA fusion model comprises: A one-dimensional numerical branch, which adopts a gated recurrent unit to process a one-dimensional fusion feature vector composed of sensitive spectral data and colorimetric parameters; A two-dimensional implicit branch, which adopts a BiT to process the GASF image obtained in step 3; A two-dimensional explicit branch, which adopts a BiT to process the RGB image.

Images