A multimodal non-destructive method, apparatus, system and medium for detecting sugar content in fruits.

Abstract

This invention discloses a multimodal non-destructive method, apparatus, system, and medium for detecting fruit sugar content. The method includes preprocessing input near-infrared spectral images and visible light images of the fruit to obtain a one-dimensional near-infrared spectral input image tensor and a two-dimensional visible light input image tensor. A pre-trained multimodal deep learning fusion model is then used to obtain the predicted fruit sugar content. This pre-trained multimodal deep learning fusion model establishes a mapping relationship between the input near-infrared spectral input image tensor and the visible light input image tensor, and the predicted fruit sugar content. It includes a spectral feature extraction network, an image feature extraction network, a multimodal feature fusion network, and a regression prediction network. This invention aims to address the problem that single spectral information is easily affected by factors such as noise, lighting conditions, peel thickness, and water content, leading to limited prediction accuracy, and to improve the accuracy of non-destructive detection of fruit sugar content.

Description

Technical Field

[0001] This invention belongs to the field of agricultural product quality testing and intelligent sensing technology, specifically relating to a multimodal non-destructive testing method, device, system and medium for fruit sugar content. Background Technology

[0002] Sugar content in fruit is a crucial parameter for assessing fruit maturity, commercial grade, and flavor quality. Currently, fruit sugar content testing primarily relies on destructive refractometer methods, which require cutting the fruit to measure the refractive index of the juice. This method suffers from drawbacks such as sample destruction, low efficiency, and high labor costs. Near-infrared spectroscopy, on the other hand, can infer the internal chemical composition of fruit through spectral absorption characteristics, making it a typical non-destructive testing method. However, single spectral information is easily affected by factors such as noise, lighting conditions, peel thickness, and water content, resulting in limited prediction accuracy. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a multimodal non-destructive detection method, device, system and medium for fruit sugar content, which addresses the above-mentioned problems of the prior art. The present invention aims to solve the problem that single spectral information is easily affected by factors such as noise, light conditions, peel thickness and water content, resulting in limited prediction accuracy, and improve the accuracy of non-destructive detection of fruit sugar content.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0005] A multimodal, non-destructive method for detecting sugar content in fruits, comprising the following steps:

[0006] S101, preprocess the input fruit near-infrared spectrum image to obtain a one-dimensional near-infrared spectrum input image tensor, and preprocess the input fruit visible light image to extract a two-dimensional visible light input image tensor.

[0007] S102, the near-infrared spectral input image tensor and the visible light input image tensor are used to obtain the predicted fruit sugar content using a pre-trained multimodal deep learning fusion model. The multimodal deep learning fusion model is pre-trained to establish a mapping relationship between the input near-infrared spectral input image tensor and the visible light input image tensor, and the predicted fruit sugar content. The multimodal deep learning fusion model includes: a spectral feature extraction network for extracting high-dimensional spectral fingerprint features from the one-dimensional near-infrared spectral input image tensor; an image feature extraction network for extracting image spatial texture features from the two-dimensional visible light input image tensor; a multimodal feature fusion network for aligning and weighting the heterogeneous features composed of the high-dimensional spectral fingerprint features and the image spatial texture features through a cross-attention mechanism to generate fused features; and a regression prediction network for performing sugar content regression prediction on the fused features to obtain the predicted fruit sugar content.

[0008] Optionally, the preprocessing of the input near-infrared spectral image of the fruit in step S101 to obtain the near-infrared spectral input image tensor includes:

[0009] S201 converts the spectral intensity vector in the input near-infrared spectral image of the fruit into absorbance;

[0010] S202, the absorbance is smoothed and denoised to remove high-frequency random noise and enhance the spectral peak characteristics;

[0011] S203, the absorbance after smoothing and denoising is normally transformed to obtain a one-dimensional infrared spectrum input image tensor;

[0012] Step S101, which involves preprocessing the input visible light image of the fruit to extract the visible light input image tensor, includes:

[0013] S301, The Otsu thresholding method is used to binarize the R channel of the input fruit visible light image to extract the fruit mask;

[0014] S302, set the background pixels of the input fruit visible light image to zero according to the fruit mask;

[0015] S303 performs pixel-level normalization on each channel of the visible light image of the fruit after setting the background pixels to zero to obtain a two-dimensional visible light input image tensor.

[0016] Optionally, the spectral feature extraction network includes three cascaded feature extraction layers and a global average pooling layer. The first feature extraction layer uses a one-dimensional convolutional layer with a kernel size of 5 and a number of kernels of 32 to capture local wide-band features in the spectrum. The second feature extraction layer uses a one-dimensional convolutional layer with a kernel size of 7 and a number of channels of 64 to capture a larger receptive field that helps extract cross-band correlation features. The third feature extraction layer uses a one-dimensional convolutional layer with a kernel size of 3 and a number of channels of 128 to capture fine high-frequency spectral features and increase the feature dimension. Each convolutional operation in the three feature extraction layers is followed by a batch normalization layer and a ReLU activation function to accelerate convergence and introduce nonlinearity. l The expression for the calculation function of the output feature map of the k-th convolutional kernel of the feature extraction layer is:

[0017] ;

[0018] in, For the first l The k-th convolutional kernel of the feature extraction layer outputs the feature value at position i in the feature map. It is the ReLU activation function. For the first l-The number of channels in a feature extraction layer For the first l The kernel size of each feature extraction layer For the first l The weight of the k-th convolutional kernel in the c-th channel of a feature extraction layer at position j. For the first l- Position of the c-th channel in the output feature map of a feature extraction layer eigenvalues, For the first l The bias term of the kth convolutional kernel of the third feature extraction layer; the global average pooling layer is used to compress the output feature map of the third feature extraction layer into a 1×128 feature vector. This serves as the obtained high-dimensional spectral fingerprint feature.

[0019] Optionally, the image feature extraction network includes an initial convolutional layer, a max pooling layer, four cascaded residual feature extraction modules, a global average pooling layer, and a flattening layer. The output features of the initial convolutional layer are processed by batch normalization and ReLU activation function and then used as the input features of the max pooling layer. The residual feature extraction module includes two levels of sub-residual feature extraction modules. Each sub-residual feature extraction module includes two 3×3 convolutional blocks. The output features of the first 3×3 convolutional block are processed by batch normalization and ReLU activation function and then used as the input features of the second 3×3 convolutional block. The input features are obtained by batch normalizing the second 3×3 convolutional block and concatenating it with the input features of the first 3×3 convolutional block, then processing it with the ReLU activation function to serve as the output features of the sub-residual feature extraction module. The image spatial texture features are obtained by globally averaging and flattening the output features of the last residual feature extraction module. The first sub-residual feature extraction module of the last three residual feature extraction modules contains an additional 1×1 convolutional block for downsampling, and its function expression is:

[0020] ;

[0021] in, For a 1×1 convolutional block, the function expressions for the other sub-residual feature extraction modules are:

[0022] ;

[0023] in, For the first The intermediate output features of the sub-residual feature extraction module These are two 3×3 convolutional blocks for the sub-residual feature extraction module. For the first The input features of the sub-residual feature extraction module For the first Trainable weight parameters for the sub-residual feature extraction module For activation function, It is a 1×1 convolutional block. This is the sequence number of the sub-residual feature extraction module.

[0024] Optionally, the multimodal feature fusion network includes:

[0025] The feature tokenization layer is used to convert heterogeneous features composed of high-dimensional spectral fingerprint features and image spatial texture features into serialized token forms:

[0026] , ;

[0027] in, High-dimensional spectral fingerprint features in token form. High-dimensional spectral fingerprint features Location encoding for learnable high-dimensional spectral fingerprint features, Image spatial texture features in token form, For image spatial texture features, Location encoding of learnable image spatial texture features;

[0028] The intramodal self-attention layer is used to compute self-attention within their respective modalities for token-based high-dimensional spectral fingerprint features and image spatial texture features, thereby obtaining context-enhanced feature sequences and ultimately, context-enhanced spectral dimension feature sequences. and image dimensional feature sequences ;

[0029] A bidirectional cross-attention layer is used to target context-enhanced spectral dimension feature sequences. and image dimensional feature sequences Two parallel cross-attention mechanisms are used to achieve bidirectional challenge and response between modalities, thereby obtaining the cross-modal enhanced spectral dimension features. and image dimensional features ;

[0030] The gate coefficient calculation layer is used to calculate the spectral dimension features after cross-modal enhancement. and image dimensional features Calculate two gating coefficients using a fully connected layer:

[0031] ;

[0032] ;

[0033] in, and These are the gate coefficients for the spectral dimension and the image dimension, respectively. For activation function, and Weighting coefficients;

[0034] A gated weighted fusion layer is used to combine cross-modal enhanced spectral dimension features. and image dimensional features The final fused feature is obtained by weighting the data using two gating coefficients:

[0035] ;

[0036] in, This is a feature of fusion.

[0037] Optionally, the regression prediction network includes:

[0038] The feature aggregation layer is used to compress the fused features along the sequence length dimension using global average pooling, in order to extract the average response of the entire feature sequence and obtain the global feature vector. ;

[0039] Regression networks are used to process global feature vectors. A sugar content regression prediction is performed to obtain the sugar content of the fruit. The regression network includes three cascaded fully connected layers. A non-linear activation function and a random dropout layer are introduced between adjacent fully connected layers. The first fully connected layer is used to compress the 256-dimensional global features to 128-dimensional features. The second fully connected layer further reduces the feature dimension from 128-dimensional to 32-dimensional features. The third fully connected layer is the output layer, which is used to map the 32-dimensional features to a 1-dimensional scalar as the obtained predicted fruit sugar content.

[0040] Optionally, the loss function used during training of the multimodal deep learning fusion model has the following expression:

[0041] ;

[0042] ;

[0043] ;

[0044] ;

[0045] ;

[0046] in, For loss function, ~ For balance coefficient, For robust regression losses, For Pearson correlation coefficient loss, The loss is due to gating regularization constraints. This refers to the batch sample size. The first The actual sugar content of the fruit in each sample For the first Predicted fruit sugar content for each sample, Huber loss function. For Huber threshold, and These represent the average of the actual and predicted sugar content of the fruit in the batch samples, respectively. and These are the gating coefficients for the spectral dimension and the image dimension, respectively.

[0047] The present invention also provides a fruit sugar content non-destructive testing device for applying the multimodal fruit sugar content non-destructive testing method described above, comprising a dark box, an electric turntable provided on the top of the inner side of the dark box, a C-shaped track mounted on the electric turntable, a camera unit mounted on the C-shaped track, the camera unit comprising a light source, a near-infrared spectrometer for acquiring near-infrared spectral images of the fruit, and a visible light camera for acquiring visible light images of the fruit, a hollow base provided on the bottom of the inner side of the dark box, a black silicone tray provided on the top of the hollow base, an opening provided on the bottom of the black silicone tray, an optical fiber and a ring-shaped supplementary light provided inside the hollow base, the end of the optical fiber being located at the opening.

[0048] The present invention also provides a multimodal non-destructive testing system for fruit sugar content, comprising a microprocessor and a memory interconnected thereto, wherein the microprocessor is programmed or configured to execute the multimodal non-destructive testing method for fruit sugar content.

[0049] The present invention also provides a computer-readable storage medium storing a computer program or instructions that are programmed or configured to execute the multimodal non-destructive fruit sugar content detection method by a processor.

[0050] Compared with existing technologies, the present invention mainly achieves the following beneficial effects: The present invention proposes a non-destructive detection method for fruit sugar content based on multimodal fusion. It fuses near-infrared spectral images and visible light RGB images of the fruit for inference. Near-infrared spectral images can realize the spectral characteristics of the fruit's internal maturity, while visible light images can provide visual characteristics reflecting the fruit's external maturity, such as peel color, texture, and surface condition. These features complement the spectral information. The multimodal fusion model can integrate the internal components and external characterization information of the fruit, improving the accuracy and generalization ability of sugar content prediction. It can solve the problem that single spectral information is easily affected by factors such as noise, light conditions, peel thickness, and water content, resulting in limited prediction accuracy. It achieves rapid, stable, and high-precision prediction of fruit sugar content and can be deployed on handheld devices for greater convenience. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of the basic process of the method in an embodiment of the present invention.

[0052] Figure 2 The diagram below is a schematic diagram of the preprocessing process in an embodiment of the present invention, wherein (a) is a flowchart of the preprocessing of near-infrared spectral images of fruits, and (b) is a flowchart of the preprocessing of visible light images of fruits.

[0053] Figure 3 This is a schematic diagram of the network structure of the spectral feature extraction network in an embodiment of the present invention.

[0054] Figure 4 This is a schematic diagram of the network structure of the image feature extraction network in an embodiment of the present invention.

[0055] Figure 5 This is a schematic diagram of the network structure of the multimodal feature fusion network in an embodiment of the present invention.

[0056] Figure 6 This is a schematic diagram of the network structure of the regression prediction network in an embodiment of the present invention.

[0057] Figure 7 The diagram below is a schematic representation of the device according to an embodiment of the present invention, and the illustrations are as follows: 1. Dark box; 2. Electric turntable; 3. C-shaped track; 4. Camera unit; 5. Hollow base; 6. Black silicone tray; 7. Optical fiber; 8. Ring light. Detailed Implementation

[0058] To enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions of the present invention will be further described in detail below with reference to the accompanying drawings in the embodiments of the present invention.

[0059] like Figure 1 As shown, the multimodal non-destructive fruit sugar content detection method of this embodiment includes the following steps:

[0060] S101, preprocess the input fruit near-infrared spectrum image to obtain a one-dimensional near-infrared spectrum input image tensor, and preprocess the input fruit visible light image to extract a two-dimensional visible light input image tensor.

[0061] S102, the near-infrared spectral input image tensor and the visible light input image tensor are used to obtain the predicted fruit sugar content using a pre-trained multimodal deep learning fusion model. The multimodal deep learning fusion model is pre-trained to establish a mapping relationship between the input near-infrared spectral input image tensor and the visible light input image tensor, and the predicted fruit sugar content. The multimodal deep learning fusion model includes: a spectral feature extraction network for extracting high-dimensional spectral fingerprint features from the one-dimensional near-infrared spectral input image tensor; an image feature extraction network for extracting image spatial texture features from the two-dimensional visible light input image tensor; a multimodal feature fusion network for aligning and weighting the heterogeneous features composed of the high-dimensional spectral fingerprint features and the image spatial texture features through a cross-attention mechanism to generate fused features; and a regression prediction network for performing sugar content regression prediction on the fused features to obtain the predicted fruit sugar content.

[0062] The preprocessing in step S101 is used to eliminate physical noise and baseline drift.

[0063] like Figure 2 As shown in (a), step S101, which involves preprocessing the input near-infrared spectral image of the fruit to obtain the near-infrared spectral input image tensor, includes:

[0064] S201 converts the spectral intensity vector in the input near-infrared spectral image of the fruit into absorbance;

[0065] The spectral intensity vector in a near-infrared spectral image of a fruit can be represented as: ,in This refers to the number of wavelength bands. Since the sugar concentration inside the fruit has a linear relationship with the degree of light absorption, rather than a direct linear relationship with the intensity of reflected light, the spectral intensity vector (reflectivity) must be converted to absorbance. A All subsequent smoothing, denoising, and deep learning feature extraction are based on absorbance data. A The process involves converting reflectance / transmittance intensity to absorbance while adhering to the linear assumption of the Lambert-Beer law; the functional expression is as follows:

[0066] ;

[0067] Where R is the reflectivity, The spectral intensity is that of a standard white board (or an unloaded, fully transparent board). The dark current noise was measured after the light source was turned off. The original intensity of the fruit sample represents the original light intensity count value transmitted through or reflected from the surface of the fruit under test when the spectrometer is in the illumination state.

[0068] S202, the absorbance is smoothed and denoised to remove high-frequency random noise and enhance the spectral peak characteristics;

[0069] As an optional implementation, the SG convolution smoothing algorithm is used to remove high-frequency random noise and enhance spectral peak features. For the _th_ ... wavelength points its smoothed value The expression for the calculation function is as follows:

[0070] ;

[0071] in: The size of the sliding window (11 in this embodiment); The convolution coefficients are the least squares fitting polynomials. This is the normalization factor. To eliminate baseline drift, the first derivative can be calculated, and the coefficient set corresponding to the first derivative is used. .

[0072] S203, the absorbance after smoothing and denoising is subjected to a normal transformation to obtain a one-dimensional infrared spectral input image tensor; to eliminate the overall spectral tilt and intensity shift caused by differences in fruit size and surface scattering coefficients, each spectrum is normalized using SNV (Standard Normal Variable), i.e., a normal transformation. For a spectral vector Its normal transformation function expression is:

[0073] ;

[0074] in, for The result of the normal transformation, the spectral mean and standard deviation The definition of is:

[0075] ;

[0076] After normal transformation, all spectral data will have zero mean and unit variance, which makes the model focus more on the relative shape of the peaks rather than their absolute intensity.

[0077] like Figure 2 As shown in (b), step S101, which involves preprocessing the input fruit visible light image to extract the visible light input image tensor, includes:

[0078] S301, The Otsu thresholding method is used to binarize the R channel of the input fruit visible light image to extract the fruit mask;

[0079] The input visible light image of the fruit can be represented as: The R channel was binarized using the Otsu thresholding method to extract the fruit mask. M .

[0080] S302, set the background pixels of the input fruit visible light image to zero according to the fruit mask; this can be represented as:

[0081] ;

[0082] in, For position Pixel value at that location, The input visible light image of the fruit is located at... Pixel value at that location, Masking fruit M In position The pixel values ​​at that location are then determined. A rectangular area containing only the fruit is then cropped based on the mask boundaries and scaled to a fixed size (×224).

[0083] S303, after setting the background pixels to zero, performs pixel-level normalization on each channel of the visible light image of the fruit to obtain a two-dimensional visible light input image tensor. To accelerate the convergence of the deep learning model and prevent gradient vanishing or exploding, the following formula is used to perform pixel-level normalization on each channel (R, G, B) of the input image:

[0084] ;

[0085] in, This represents the normalized pixel value of each channel of the original image, ranging from [0,1]. The input image tensor is the data that is ultimately fed into the image feature extraction network. The mean of the dataset is taken as [0.485, 0.456, 0.406] in this embodiment. The standard deviation of the dataset is [0.229, 0.224, 0.225] in this embodiment.

[0086] like Figure 3 As shown, the spectral feature extraction network consists of three cascaded feature extraction layers and one global average pooling layer. Figure 3The numbers above the arrows represent the feature sizes. For example, 1×256 indicates that the input feature dimension is 1×256. The first feature extraction layer uses a one-dimensional convolutional layer with a kernel size of 5 (Kernel=5) and 32 filters to capture local wide-band features in the spectrum. The second feature extraction layer uses a one-dimensional convolutional layer with a kernel size of 7 and 64 channels to capture a larger receptive field, which helps extract cross-band associated features. The third feature extraction layer uses a one-dimensional convolutional layer with a kernel size of 3 and 128 channels to capture fine high-frequency spectral features and increase the feature dimension. Each convolutional operation in the three feature extraction layers is followed by a batch normalization (BN) layer and a ReLU activation function to accelerate convergence and introduce nonlinearity. (The last sentence appears to be incomplete and possibly refers to a specific layer or function.) l The expression for the calculation function of the output feature map of the k-th convolutional kernel of the feature extraction layer is:

[0087] ;

[0088] in, For the first l The k-th convolutional kernel of the feature extraction layer outputs the feature value at position i in the feature map. The ReLU activation function is expressed as follows: , For the first l- The number of channels in a feature extraction layer For the first l The kernel size of each feature extraction layer For the first l The weight of the k-th convolutional kernel in the c-th channel of a feature extraction layer at position j. For the first l- Position of the c-th channel in the output feature map of a feature extraction layer eigenvalues, For the first l The bias term of the k-th convolutional kernel in each feature extraction layer; after three convolutional extraction layers, to reduce the number of model parameters and prevent overfitting, this embodiment abandons the traditional fully connected layer unwinding method and instead adopts Global Average Pooling (GAP). The global average pooling layer is used to compress the output feature map of the third feature extraction layer into a 1×128 feature vector. The obtained high-dimensional spectral fingerprint features can be represented as:

[0089] ;

[0090] in, For feature vectors The output of the k-th convolutional kernel, The k-th convolutional kernel of the last feature extraction layer outputs the feature value at position i in the feature map. This represents the number of positions. Feature vector. It is invariant to spectral shifts and can be used as input for subsequent multimodal fusion modules.

[0091] The image feature extraction network employs a deep convolutional neural network based on a residual learning framework. The input data consists of geometrically corrected and normalized RGB image tensors. To address the vanishing gradient problem in deep networks and extract features at higher semantic levels, such as... Figure 4 As shown, the image feature extraction network includes an initial convolutional layer, a max pooling layer, and four cascaded residual feature extraction modules. Figure 4 The dashed box in the middle represents a residual feature extraction module, a global average pooling and flattening layer, with input features of dimension 224×224×3. The output features of the initial convolutional layer are processed by batch normalization and ReLU activation function and then used as input features for the max pooling layer. The residual feature extraction module includes two levels of sub-residual feature extraction modules. Each sub-residual feature extraction module includes two 3×3 convolutional blocks. The output features of the first 3×3 convolutional block are processed by batch normalization and ReLU activation function and then used as input features for the second 3×3 convolutional block. The input features of the second 3×3 convolutional block are batch normalized and concatenated with the input features of the first 3×3 convolutional block, then processed by ReLU activation function, and then used as the output features of the sub-residual feature extraction module. The image spatial texture features obtained by the output features of the last residual feature extraction module are global average pooling and flattening layer, and their dimension is 1×512. See also Figure 4 The first sub-residual feature extraction module of the latter three of the four residual feature extraction modules contains an additional 1×1 convolutional block for downsampling, and its function expression is:

[0092] ;

[0093] in, For a 1×1 convolutional block, the function expressions for the other sub-residual feature extraction modules are:

[0094] ;

[0095] in, For the first The intermediate output features of the sub-residual feature extraction module These are two 3×3 convolutional blocks for the sub-residual feature extraction module. For the first The input features of the sub-residual feature extraction module For the first Trainable weight parameters for the sub-residual feature extraction module For activation function, It is a 1×1 convolutional block. This is the sequence number of the sub-residual feature extraction module.

[0096] Through the above mechanism, the network can extract the following image spatial texture features highly correlated with sugar content layer by layer. These features include: Color features: responding to hue changes on the fruit peel surface through shallow convolution, reflecting the degree of chlorophyll degradation and anthocyanin accumulation (i.e., maturity). Texture features: capturing the roughness and pore distribution of the fruit peel surface, indirectly reflecting the fruit's water tension. Morphology and defects: identifying fruit fullness and the presence of spots or damage on the surface through deep features; these physical properties typically have a statistically significant covariance relationship with the fruit's internal metabolic level and sugar distribution.

[0097] After feature extraction by four residual feature extraction modules, the output feature map typically has a dimension of 7×7×512. To achieve modal alignment with spectral features and preserve global spatial information, global average pooling is also employed.

[0098] ;

[0099] in, This represents the global pooling feature for channel c. For the height and width of the image, The position in the output features of the last residual feature extraction module The feature values ​​at that location are then flattened to obtain a 1×512 dimensional image space texture feature. The spatial texture features of this image highly condense the external visual phenotypic information of the fruit.

[0100] In this embodiment, the multimodal feature fusion network is a bidirectional interactive architecture based on Transformer. For example... Figure 5 As shown, the multimodal feature fusion network includes:

[0101] The feature tokenization layer (omitted in the diagram) is used to convert heterogeneous features composed of high-dimensional spectral fingerprint features and image spatial texture features into serialized token forms:

[0102] , ;

[0103] in, High-dimensional spectral fingerprint features in token form. High-dimensional spectral fingerprint features Location encoding for learnable high-dimensional spectral fingerprint features, Image spatial texture features in token form, For image spatial texture features, Location encoding of learnable image spatial texture features;

[0104] The intramodal self-attention layer is used to compute self-attention within their respective modalities for token-based high-dimensional spectral fingerprint features and image spatial texture features, thereby obtaining context-enhanced feature sequences and ultimately, context-enhanced spectral dimension feature sequences. and image dimensional feature sequences To enhance the global contextual features within each modality before modal interaction, the data first passes through two independent self-attention modules in parallel. Taking the spectral branch as an example, the input... generate Q , K , V vector:

[0105] ;

[0106] in, , and These are the query matrix, key matrix, and value matrix for the spectral modes, respectively. , and These are the weight matrices corresponding to the query matrix, key matrix, and value matrix, respectively. The function expression for the self-attention module is:

[0107] ;

[0108] in, The self-attention features obtained from the calculation constitute the context-enhanced feature sequence, thus yielding the context-enhanced spectral dimension feature sequence. and image dimensional feature sequences These represent spectral features that aggregate global band information and visual features that aggregate global spatial information, respectively. The dimension of the key matrix is ​​given. These features are then input into two parallel cross-attention streams to achieve bidirectional challenge and response between modalities.

[0109] A bidirectional cross-attention layer is used to target context-enhanced spectral dimension feature sequences. and image dimensional feature sequences Two parallel cross-attention mechanisms are used to achieve bidirectional challenge and response between modalities, thereby obtaining the cross-modal enhanced spectral dimension features. and image dimensional features Taking the spectral branch as an example again, the calculation formula is:

[0110] ;

[0111] in, , and These are the weight matrices corresponding to the query matrix, key matrix, and value matrix in the cross-attention mechanism, respectively.

[0112] The gate coefficient calculation layer is used to calculate the spectral dimension features after cross-modal enhancement. and image dimensional features Calculate two gating coefficients using a fully connected layer:

[0113] ;

[0114] ;

[0115] in, and These are the gate coefficients for the spectral dimension and the image dimension, respectively. For activation function, and Weighting coefficients;

[0116] A gated weighted fusion layer is used to combine cross-modal enhanced spectral dimension features. and image dimensional features The final fused feature is obtained by weighting the data using two gating coefficients:

[0117] ;

[0118] in, To facilitate feature fusion, considering the varying confidence levels of spectral and image data in different detection scenarios (e.g., image weights should be reduced when there is strong ambient light interference), this invention introduces an adaptive gating mechanism for final fusion. The system learns two gating coefficients through a fully connected layer. and ( ), final fusion features The calculation is a weighted sum, and this fusion feature... It contains high-level features that have undergone bidirectional semantic alignment and adaptive weighting, and is then fed into the regression prediction head to output the predicted sugar content value.

[0119] like Figure 6 As shown, the regression prediction network includes:

[0120] The feature aggregation layer is used to compress the fused features along the sequence length dimension using global average pooling (GAP) to extract the average response of the entire feature sequence and obtain the global feature vector. , can be represented as:

[0121] ;

[0122] in, As a feature of fusion, This indicates that the first i bits are truncated.

[0123] Regression networks are used to process global feature vectors. A sugar content regression prediction method is used to obtain the sugar content of the fruit. The regression network consists of three cascaded fully connected layers (FC). Non-linear activation functions and random dropout layers are introduced between adjacent fully connected layers. The first fully connected layer compresses the 256-dimensional global features to 128 dimensions. The second fully connected layer further reduces the feature dimension from 128 to 32 dimensions. The third fully connected layer is the output layer, which maps the 32-dimensional features to a 1-dimensional scalar as the predicted fruit sugar content. The first fully connected layer can be represented as:

[0124] ;

[0125] in, The output features of the first fully connected layer, This is the weight matrix. This is the ReLU activation function. For random drop-out layers, As a bias, the random dropout ratio of the random dropout layer is set to 0.5 to randomly drop some neuron connections, enhancing the model's robustness. The second fully connected layer can be represented as:

[0126] ;

[0127] in, The output features of the second fully connected layer, For bias, The weight matrix is ​​used; the third fully connected layer is the output layer, which maps the 32-dimensional features to a 1-dimensional scalar, thus predicting the sugar content of the fruit. :

[0128] ;

[0129] in, This is the weight matrix. As a bias, no activation function is used here to allow the output to be a continuous real value in any range.

[0130] As an optional implementation, the multimodal feature fusion network can use a hybrid MSE loss function during training. The functional expression of the hybrid MSE loss function is as follows:

[0131] ;

[0132] ;

[0133] in, For the mixed MSE loss function, The regularization coefficient is . Mean square error, For the network parameters of the multimodal deep learning fusion model, for The L1 norm, For the sample size, The first The actual sugar content of the fruit in each sample For the first The predicted sugar content of each sample of fruit. As an alternative implementation, to address the label noise problem caused by fruit non-uniformity and enhance the model's sensitivity to sugar content trends, the multimodal feature fusion network can employ a multi-task hybrid loss function based on trend constraints and robust regression during training. This multi-task hybrid loss function is composed of a weighted robust regression component (robust regression loss), a trend consistency component (Pearson correlation coefficient loss), and a gated sparsity component (gated regularization constraint loss). The specific calculation function expression is as follows:

[0134] ;

[0135] ;

[0136] ;

[0137] ;

[0138] ;

[0139] in, This is the loss function (i.e., the multi-task hybrid loss function). ~ For balance coefficient, For robust regression losses, For Pearson correlation coefficient loss, The loss is due to gating regularization constraints. This refers to the batch sample size. The first The actual sugar content of the fruit in each sample For the first Predicted fruit sugar content for each sample Here is the Huber loss function. For Huber threshold, and These represent the average of the actual and predicted sugar content of the fruit in the batch samples, respectively. and These are the gate coefficients for the spectral dimension and the image dimension, respectively. In this embodiment, the Huber threshold... Set to 1.0. Balance coefficient. The values ​​are 1.0, 0.5, and 0.01.

[0140] To verify the effectiveness of the multimodal non-destructive fruit sugar content detection method in this embodiment, comparative and ablation experiments were conducted, and the results are shown in Table 1 and Table 2, respectively.

[0141] Table 1: Comparative Experiment Results

[0142]

[0143] Table 1 compares the sugar content prediction performance of different methods. PLS and ResNet-18 are existing methods, while SpecNet is the subnet (spectral branch) of this embodiment. The evaluation metrics used are R² (coefficient of determination) and RMSE (root mean square error), two of the most commonly used evaluation metrics in regression models. As shown in Table 1, using RGB images alone yields the worst results, indicating that relying solely on appearance is insufficient for accurately detecting internal sugar content, although it does contain some relevant information (such as maturity). Using the spectral branch (SpecNet) of this embodiment alone yields better results, indicating that near-infrared spectroscopy is the dominant factor in sugar content detection. The method in this embodiment achieves the best results, demonstrating that by fusing internal spectral component information with external visual representation information, the model can achieve complementarity, thereby overcoming the accuracy bottleneck of single-modal detection.

[0144] Table 2: Ablation Experiment Results

[0145]

[0146] Table 2 shows the ablation experiment results of the key improved modules. Here, "attention" refers to the bidirectional cross-attention layer in the multimodal feature fusion network, and "gating" refers to the gating coefficient calculation layer + gating weighted fusion layer. Compared to simple direct concatenation, the introduction of the bidirectional cross-attention layer improved the R² (coefficient of determination) by 0.025, indicating that the intermodal interaction of the bidirectional cross-attention layer in the multimodal feature fusion network achieves semantic alignment between spectral features and image features, enhancing the expressive power of the features. Model D further reduced the RMSE (root mean square error) compared to Model B. This indicates that the gating coefficient calculation layer + gating weighted fusion layer can automatically suppress the negative impact of low-quality modalities (such as illumination interference or spectral noise) based on different samples, improving the robustness of the system. Model D uses a hybrid MSE loss function, which is improved to a single loss function. This method can suppress physical noise in the training data and enhance the model's ability to fit the trend of sugar content changes. Therefore, the multimodal non-destructive fruit sugar content detection method in this embodiment can fuse near-infrared spectral images and visible light RGB images of fruits for inference, thereby achieving rapid and non-destructive prediction of fruit sugar content.

[0147] like Figure 7 As shown, this embodiment also provides a fruit sugar content non-destructive testing device for a multimodal fruit sugar content non-destructive testing method, including an electric turntable 2 located on the top of the inner side of a dark box 1, a C-shaped track 3 mounted on the electric turntable 2, and a camera unit 4 mounted on the C-shaped track 3 that can move along the C-shaped track 3. The camera unit 4 includes a light source and a sensor for acquiring fruit sugar content (…). Figure 7As shown in Figure A, a near-infrared spectrometer for near-infrared spectral imaging and a visible light camera for acquiring visible light images of the fruit are used. The bottom of the dark box 1 has a hollow base 5, and the top of the hollow base 5 has a black silicone tray 6. The bottom of the black silicone tray 6 has an opening. An optical fiber 7 and a ring-shaped supplementary light 8 are located inside the hollow base 5, with the end of the optical fiber 7 located at the opening. In this embodiment, the dimensions of the dark box 1 are 400×400×600mm. The electric turntable 2 and the C-shaped track 3 are existing actuators that can carry the camera unit 4 to rotate and tilt the fruit for imaging. The camera unit 4 includes a light source (white LED dome light source), a near-infrared spectrometer (wavelength range 900-1700nm) for acquiring near-infrared spectral images of the fruit, and a visible light camera (5-megapixel CMOS camera) for acquiring visible light images of the fruit. The hollow base 5 is a cylindrical hollow base machined from aluminum alloy using CNC machining, with a blackened interior. The black silicone tray 6 is made of food-grade silicone (Shore hardness 40A) and is bowl-shaped to accommodate fruits with different curvatures (apples, pears, peaches, etc.). The black silicone tray 6 has a 20mm diameter through-hole in the center. When fruit is placed on it, the soft silicone deforms and conforms to the fruit peel, forming a perfect light seal and preventing light from the light source from directly entering the probe without passing through the fruit pulp. The optical fiber 7 uses a single fiber with a core diameter of 600mm. The quartz optical fiber is located at the center of the through hole in the black silicone tray 6, with its end face slightly lower than the silicone surface of the tray 6 by 1 mm to avoid scratching the fruit peel. Light enters the fruit and, after being scattered by the internal tissues, photons carrying sugar content information return and are transmitted to the near-infrared spectrometer via the optical fiber. The ring-shaped supplemental light 8 consists of four 50W halogen lamps (color temperature 3100K) evenly distributed around the probe of the optical fiber 7, symmetrically illuminating the through hole in a V-shape. The light is focused at a 45-degree angle upwards onto the fruit peel above the through hole.

[0148] Furthermore, this embodiment also provides a multimodal non-destructive fruit sugar content detection system, including a microprocessor and a memory interconnected, wherein the microprocessor is programmed or configured to execute the multimodal non-destructive fruit sugar content detection method. The present invention also provides a computer-readable storage medium storing a computer program or instructions programmed or configured to execute the multimodal non-destructive fruit sugar content detection method via a processor.

[0149] Those skilled in the art will understand that the technical solutions provided by this invention may take the form of a method, system, or computer program product. Therefore, this invention may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this invention may take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce an implementation of the flowchart... Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0150] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A multimodal, non-destructive method for detecting sugar content in fruits, characterized in that, Includes the following steps: S101, preprocess the input fruit near-infrared spectrum image to obtain a one-dimensional near-infrared spectrum input image tensor, and preprocess the input fruit visible light image to extract a two-dimensional visible light input image tensor. S102, the near-infrared spectral input image tensor and the visible light input image tensor are used to obtain the predicted fruit sugar content using a pre-trained multimodal deep learning fusion model. The multimodal deep learning fusion model is pre-trained to establish a mapping relationship between the input near-infrared spectral input image tensor and the visible light input image tensor, and the predicted fruit sugar content. The multimodal deep learning fusion model includes: a spectral feature extraction network for extracting high-dimensional spectral fingerprint features from the one-dimensional near-infrared spectral input image tensor; an image feature extraction network for extracting image spatial texture features from the two-dimensional visible light input image tensor; a multimodal feature fusion network for aligning and weighting the heterogeneous features composed of the high-dimensional spectral fingerprint features and the image spatial texture features through a cross-attention mechanism to generate fused features; and a regression prediction network for performing sugar content regression prediction on the fused features to obtain the predicted fruit sugar content. The spectral feature extraction network comprises three cascaded feature extraction layers and a global average pooling layer. Each of the three feature extraction layers is followed by a batch normalization layer and a ReLU layer after a convolution operation. The activation function accelerates convergence and introduces nonlinearity. The image feature extraction network includes an initial convolutional layer, a max pooling layer, four cascaded residual feature extraction modules, a global average pooling and flattening layer, and a residual feature extraction module comprising two sub-residual feature extraction modules. Each sub-residual feature extraction module includes two 3×3 convolutional blocks. The output features of the first 3×3 convolutional block are processed by batch normalization and ReLU activation function and used as the input features of the second 3×3 convolutional block. The input features of the second 3×3 convolutional block are concatenated with the input features of the first 3×3 convolutional block after batch normalization and then processed by ReLU activation function to become the output features of the sub-residual feature extraction module. The image spatial texture features are obtained by global average pooling and flattening the output features of the last residual feature extraction module after global average pooling and flattening layer. The first sub-residual feature extraction module of the last three residual feature extraction modules contains an additional 1×1 convolutional block for downsampling, and its function expression is: ； in, For a 1×1 convolutional block, the function expressions for the other sub-residual feature extraction modules are: ； in, For the first The intermediate output features of the sub-residual feature extraction module These are two 3×3 convolutional blocks for the sub-residual feature extraction module. For the first The input features of the sub-residual feature extraction module For the first Trainable weight parameters for the sub-residual feature extraction module For activation function, It is a 1×1 convolutional block. This is the sequence number of the sub-residual feature extraction module.

2. The multimodal, non-destructive method for detecting fruit sugar content according to claim 1, characterized in that, Step S101 involves preprocessing the input near-infrared spectral image of the fruit to obtain the near-infrared spectral input image tensor, including: S201 converts the spectral intensity vector in the input near-infrared spectral image of the fruit into absorbance; S202, the absorbance is smoothed and denoised to remove high-frequency random noise and enhance the spectral peak characteristics; S203, the absorbance after smoothing and denoising is normally transformed to obtain a one-dimensional infrared spectrum input image tensor; Step S101, which involves preprocessing the input visible light image of the fruit to extract the visible light input image tensor, includes: S301, The Otsu thresholding method is used to binarize the R channel of the input fruit visible light image to extract the fruit mask; S302, set the background pixels of the input fruit visible light image to zero according to the fruit mask; S303 performs pixel-level normalization on each channel of the visible light image of the fruit after setting the background pixels to zero to obtain a two-dimensional visible light input image tensor.

3. The multimodal, non-destructive method for detecting fruit sugar content according to claim 1, characterized in that, In the three feature extraction layers of the spectral feature extraction network: the first feature extraction layer uses a one-dimensional convolutional layer with a kernel size of 5 and a number of kernels of 32 to capture local wide-band features in the spectrum; the second feature extraction layer uses a one-dimensional convolutional layer with a kernel size of 7 and a number of channels of 64 to capture a larger receptive field that helps to extract cross-band associated features. The third feature extraction layer uses a one-dimensional convolutional layer with a kernel size of 3 and 128 channels to capture fine high-frequency spectral features and increase the feature dimension; any third... l The expression for the calculation function of the output feature map of the k-th convolutional kernel of the feature extraction layer is: ； in, For the first l The k-th convolutional kernel of the feature extraction layer outputs the feature value at position i in the feature map. It is the ReLU activation function. For the first l- The number of channels in a feature extraction layer For the first l The kernel size of each feature extraction layer For the first l The weight of the k-th convolutional kernel in the c-th channel of a feature extraction layer at position j. For the first l- Position of the c-th channel in the output feature map of a feature extraction layer eigenvalues, For the first l The bias term of the kth convolutional kernel of the third feature extraction layer; the global average pooling layer is used to compress the output feature map of the third feature extraction layer into a 1×128 feature vector. This serves as the obtained high-dimensional spectral fingerprint feature.

4. The multimodal, non-destructive method for detecting fruit sugar content according to claim 1, characterized in that, The multimodal feature fusion network includes: The feature tokenization layer is used to convert heterogeneous features composed of high-dimensional spectral fingerprint features and image spatial texture features into serialized token forms: ， ； in, High-dimensional spectral fingerprint features in token form. High-dimensional spectral fingerprint features, Location encoding for learnable high-dimensional spectral fingerprint features, Image spatial texture features in token form, For image spatial texture features, Location encoding of learnable image spatial texture features; The intramodal self-attention layer is used to compute self-attention within their respective modalities for token-based high-dimensional spectral fingerprint features and image spatial texture features, thereby obtaining context-enhanced feature sequences and ultimately, context-enhanced spectral dimension feature sequences. and image dimensional feature sequences ; A bidirectional cross-attention layer is used to target context-enhanced spectral dimension feature sequences. and image dimensional feature sequences Two parallel cross-attention mechanisms are used to achieve bidirectional challenge and response between modalities, thereby obtaining the cross-modal enhanced spectral dimension features. and image dimensional features ; The gate coefficient calculation layer is used to calculate the spectral dimension features after cross-modal enhancement. and image dimensional features Calculate two gating coefficients using a fully connected layer: ； ； in, and These are the gate coefficients for the spectral dimension and the image dimension, respectively. For activation function, and Weighting coefficients; A gated weighted fusion layer is used to combine cross-modal enhanced spectral dimension features. and image dimensional features The final fused feature is obtained by weighting the data using two gating coefficients: ； in, This is a feature of fusion.

5. The multimodal, non-destructive method for detecting fruit sugar content according to claim 1, characterized in that, The regression prediction network includes: The feature aggregation layer is used to compress the fused features along the sequence length dimension using global average pooling, in order to extract the average response of the entire feature sequence and obtain the global feature vector. ; Regression networks are used to process global feature vectors. A sugar content regression prediction is performed to obtain the sugar content of the fruit. The regression network includes three cascaded fully connected layers. A non-linear activation function and a random dropout layer are introduced between adjacent fully connected layers. The first fully connected layer is used to compress the 256-dimensional global features to 128-dimensional features. The second fully connected layer further reduces the feature dimension from 128-dimensional to 32-dimensional features. The third fully connected layer is the output layer, which is used to map the 32-dimensional features to a 1-dimensional scalar as the obtained predicted fruit sugar content.

6. The multimodal, non-destructive method for detecting fruit sugar content according to claim 1, characterized in that, The loss function used during training of the multimodal deep learning fusion model is expressed as follows: ； ； ； ； ； in, For loss function, ~ For balance coefficient, For robust return loss, For Pearson correlation coefficient loss, The loss is due to gating regularization constraints. This refers to the batch sample size. The first The actual sugar content of the fruit in each sample For the first Predicted fruit sugar content for each sample Here is the Huber loss function. For Huber threshold, and These represent the average of the actual and predicted sugar content of the fruit in the batch samples, respectively. and These are the gating coefficients for the spectral dimension and the image dimension, respectively.

7. A non-destructive testing device for fruit sugar content using the multimodal non-destructive testing method for fruit sugar content according to any one of claims 1 to 6, characterized in that, The device includes a dark box (1), an electric turntable (2) on the top of the inner side of the dark box (1), a C-shaped track (3) on the electric turntable (2), a camera unit (4) on the C-shaped track (3) that can move along the C-shaped track (3), the camera unit (4) including a light source, a near-infrared spectrometer for acquiring near-infrared spectral images of fruit and a visible light camera for acquiring visible light images of fruit, a hollow base (5) on the bottom of the inner side of the dark box (1), a black silicone tray (6) on the top of the hollow base (5), an opening at the bottom of the black silicone tray (6), an optical fiber (7) and a ring light (8) inside the hollow base (5), and the end of the optical fiber (7) is located at the opening.

8. A multimodal, non-destructive fruit sugar content detection system, comprising a microprocessor and a memory interconnected, characterized in that, The microprocessor is programmed or configured to perform the multimodal non-destructive fruit sugar content detection method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to execute, via a processor, the multimodal non-destructive method for detecting fruit sugar content according to any one of claims 1 to 6.

Images