Camellia fruit quality intelligent detection method and system

Abstract

The application discloses a kind of based on multimodal fusion and self-supervised pre-training camellia fruit quality intelligent detection method and system, the method core is in the complementary advantages of integrating acoustic vibration, spectral analysis, electrochemical impedance three kinds of detection modal, through self-supervised pre-training mining each modal deep feature, combined with contrast learning constructs cross-modal association, realizes maturity, damage degree and other quality indexes accurate determination.The application reduces the dependence on traditional chemical analysis, avoids artificial subjective evaluation, through non-contact / micro-contact multimodal data acquisition, cross-modal feature fusion and intelligent model inference, solves the technical pain points that single modal detection generalization ability is weak, and anti-interference is poor.This method has the characteristics of fast detection speed, high accuracy, non-destructive, strong generalization ability, suitable for the quality detection of various forest fruits, provides key technical support for post-harvest sorting, cold chain logistics whole chain quality control of camellia fruit.

Description

Technical Field

[0001] This invention relates to the field of non-destructive testing and intelligent sensing technology for agricultural products. Specifically, it relates to an intelligent quality detection method for camellia oleifera fruit that integrates multimodal data of acoustic vibration, spectral analysis, and electrochemical impedance spectroscopy, combined with self-supervised pre-training and comparative learning. This method is suitable for non-destructive testing of quality indicators such as maturity and mechanical damage, and can be widely applied to large-scale and intelligent quality control scenarios for camellia oleifera fruit and various other forestry and fruit products, directly serving core industrial processes such as post-harvest sorting, warehousing and logistics, and market grading. Background Technology

[0002] Camellia oleifera fruit is an important source of woody oil in my country. The quality of its fruit mainly depends on maturity and the degree of post-harvest mechanical damage, which directly affects the oil yield, oil quality, and economic benefits. Traditional quality testing methods rely heavily on manual experience or destructive chemical analysis, resulting in inherent defects such as low efficiency, strong subjectivity, and large sample loss. These methods cannot meet the urgent needs of modern, large-scale processing lines for high-efficiency, high-consistency, and zero-loss testing.

[0003] Currently, non-destructive testing (NDT) techniques have been extensively studied, but most methods are based on a single physical mode: acoustic vibration methods analyze the vibration response of fruit products to reflect their elasticity and hardness characteristics, but are easily affected by factors such as fruit shape and peel thickness, and are not sensitive to changes in internal chemical composition; visible / near-infrared spectroscopy detects internal chemical components (such as oils and moisture) through the light absorption and reflection characteristics of substances, but has a weak response to microstructural damage; electrochemical impedance spectroscopy reflects damage and physiological changes by characterizing cell structure and membrane integrity, but lacks sufficient correlation with the macroscopic physical properties of fruits. These single-modal methods suffer from common technical bottlenecks such as limited information, poor anti-interference ability, and weak generalization.

[0004] To compensate for the limitations of single-modality learning, some studies have attempted to fuse multimodal information. However, existing fusion techniques mostly remain at the level of simple feature splicing or post-decision voting, failing to deeply explore the intrinsic connections and complementary information between different modalities at the semantic level. In addition, mainstream methods generally adopt a fully supervised learning paradigm, whose model performance heavily relies on a large amount of manually labeled data. This results in poor model adaptability when faced with samples from different varieties, origins, and seasons, and high costs for re-labeling.

[0005] Therefore, there is an urgent need in this field for an intelligent detection method and system that can automatically learn the essential features of multimodal data, achieve deep fusion of cross-modal information, and significantly reduce dependence on labeled data. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing technologies and provide an intelligent detection method and system for camellia oleifera fruit quality based on multimodal fusion and self-supervised pre-training. This method aims to automatically extract robust features from multi-source heterogeneous data through an innovative self-supervised learning framework, utilize a contrastive learning mechanism to achieve cross-modal semantic alignment and deep fusion, and ultimately construct a high-precision, highly robust, and low-annotation-dependency intelligent detection model to achieve rapid, accurate, and non-destructive detection of the maturity and damage level of camellia oleifera fruit.

[0007] To achieve the above-mentioned technical objectives, the technical solution adopted by the present invention is as follows:

[0008] A method for intelligent detection of camellia fruit quality based on multimodal fusion and self-supervised pre-training is proposed. Its core process includes five stages: multimodal data acquisition, preprocessing, self-supervised pre-training, cross-modal fusion, and intelligent judgment. Specifically, it includes the following steps:

[0009] (1) Synchronous acquisition of multimodal data: The acoustic vibration response signal, spectral data in the visible to near-infrared bands, and electrochemical impedance spectral parameters of the target camellia fruit sample are acquired simultaneously through the non-contact acoustic vibration detection module, spectral detection module and electrochemical impedance detection module.

[0010] (2) Cross-modal data preprocessing: Modality-specific preprocessing is performed on the original modal data collected in step (1);

[0011] Wavelet thresholding (such as db6 wavelet) is used to denoise acoustic vibration signals, and power spectral density is estimated through an autoregressive model to extract resonant frequency, peak amplitude and damping ratio features.

[0012] The spectral data were smoothed by Savitzky-Golay to reduce instrument noise, the effects of scattering were eliminated by standard normal variable transformation (SNV), and the characteristic wavelengths were extracted by continuous projection algorithm (SPA) to avoid information redundancy, thus achieving data dimensionality reduction while preserving chemical information.

[0013] Logarithmic transformation and normalization were performed on the electrochemical impedance data, and core features (such as R) were extracted using partial least squares (PLS). p (X, Z, θ) retain parameters that are strongly correlated with quality changes, enhancing the model's sensitivity to changes in damage and maturity.

[0014] (3) Intramodal self-supervised pre-training feature learning: A three-branch one-dimensional convolutional neural network (improved 1D-CNN) architecture is constructed, and a self-supervised pre-training strategy based on contrastive learning is adopted. Each branch independently processes one modality of data (acoustic, spectral, impedance), and a multi-scale receptive field design is introduced in the structure to adapt to the frequency and scale characteristics of different modal signals, so as to extract robust and discriminative feature representations for subsequent fusion and judgment tasks. The details are as follows:

[0015] Three independent one-dimensional convolutional neural network branches were constructed as feature encoders, respectively processing acoustic, spectral, and impedance fundamental features. Random augmentation was applied to the data for each modality, generating different augmented views of the same sample within that modality as positive sample pairs, and data from different samples as negative samples. The encoder branches were trained by optimizing the intra-modal contrastive learning loss function, enabling them to extract deep feature representations that are robust to augmentation and possess discriminative power. After training, the encoder parameters were fixed, and the acoustic deep features were extracted and output. Spectral depth features and deep characteristics of electrochemical impedance .

[0016] (4) Cross-modal feature fusion: This stage is the core innovation of the present invention, which includes two steps: feature-level fusion and decision-level fusion, and achieves semantic alignment through comparative learning.

[0017] (4.1) Cross-modal semantic alignment and feature-level fusion: based on the results obtained in step (3) , , The core task is to achieve cross-modal semantic alignment, using the input as the basis. The specific method involves calculating the cosine similarity between feature vectors of different modalities and aligning the feature vectors of different modalities of the same sample (e.g., ...). Here, i represents sample i) is constructed as a positive sample pair, and the feature vectors of any modality from different samples are constructed as negative sample pairs. By optimizing the cross-modal contrastive learning loss function (cross-modal InfoNCE contrastive loss function), the model is driven to map the features of positive sample pairs to similar positions in the common semantic space, thereby achieving semantic alignment of feature representations of the same quality state from different modalities. The aligned features are denoted as the aligned acoustic deep features. Aligned spectral depth features Deep characteristics of aligned electrochemical impedance .

[0018] Subsequently, the aligned modal features are passed through a learnable attention weight network (e.g., a two-layer fully connected neural network), which dynamically generates a set of normalized attention weights. Furthermore, a unified cross-modal joint feature vector is generated by weighted summation of aligned features. ;in, ; .

[0019] (4.2) Decision-level fusion: aligning the deep acoustic features Input to a single-modal classifier Single-modal classifier Output the preliminary quality prediction probability distribution ;aligned spectral depth features Input to a single-modal classifier Single-modal classifier Output the preliminary quality prediction probability distribution ; to align the deep features of electrochemical impedance Input to a single-modal classifier Single-modal classifier Output the preliminary quality prediction probability distribution ;Joint features across modalities Input to joint feature classifier Joint feature classifier Output the preliminary quality prediction probability distribution .

[0020] The final decision is not a simple averaging, but rather an optimized weight allocation strategy: First, a base weight is calculated based on the independent accuracy of each classifier. Second, the performance of each classifier on subsets of samples that other classifiers are prone to misclassification is analyzed using a confusion matrix, and a complementarity correction coefficient is calculated. Finally, the weights are fused. The final probability of judging the quality of camellia fruit is determined by both the basic weight and the complementarity coefficient. ;

[0021] (5) Model training and deployment: Using a sample dataset labeled with quality tags of camellia fruit (such as maturity level and degree of damage), end-to-end supervised training or fine-tuning is performed on all learnable components (classifier, attention module, decision weight) in the above fusion architecture to determine their final parameters, thereby obtaining a complete detection model that can be used for quality judgment; the trained model is deployed to the detection system to achieve automated and non-destructive intelligent quality detection of new camellia fruit samples.

[0022] As a further improvement of the present invention, in step (1), the acoustic vibration acquisition system consists of two parts: a compressed air excitation unit and a laser Doppler vibrometer. During operation, the compressed air excitation unit generates a controllable air pressure pulse (preferably 300 kPa, excitation time 200 ms) to perform non-contact transient excitation on the camellia fruit, avoiding surface damage caused by mechanical contact; the laser Doppler vibrometer collects the microscopic vibration response signal of the fruit under excitation in real time, with a signal acquisition frequency range of 50 Hz-2000 Hz, covering the typical structural resonance frequency band of the camellia fruit, and completely recording its dynamic mechanical behavior.

[0023] As a further improvement of the present invention, in step (1), the spectral acquisition system adopts a self-built visible / near-infrared spectral imaging system, equipped with a SOC-710VP spectrometer and a halogen light source, with a wavelength range of 400nm-1000nm. The system performs white plate and dark current calibration before acquisition to ensure data consistency; the spectral cubic data of the sample is obtained by push-broom scanning to achieve non-destructive detection of the distribution of chemical components (such as oil and moisture) inside the camellia fruit.

[0024] In step (1), the electrochemical impedance data acquisition is based on an LCR meter and a custom-made parallel electrode. The electrode plate touches the surface of the camellia fruit in a non-invasive manner, and an AC test signal is applied in a wide frequency range of 100Hz-5MHz. The impedance amplitude Z, phase θ, and parallel equivalent resistance R are collected simultaneously. p The system measures multiple electrical parameters, including reactance X. It supports multi-frequency scanning and automatic parameter recording. By optimizing the test voltage and frequency (preferred at 1.0V and 3.98MHz), external interference is reduced, and electrical characteristics related to cell structure and water state are extracted, enhancing the sensitivity of damage detection.

[0025] As a further improvement to the present invention:

[0026] In step (3), three independent one-dimensional convolutional neural network branches are constructed as feature encoders to process one modality of data (such as acoustic vibration signals, spectral data sequences, or electrochemical impedance parameter sequences). Each branch serves as an independent feature encoder. Each branch adopts a deep stacked improved 1D-CNN design, with the following modular structure:

[0027] (1) Input layer: Receives preprocessed modal timing data;

[0028] (2) Convolutional Layers: Composed of multiple standard convolutional modules. One-dimensional convolutional layer (Conv1D), using multiple convolutional kernels (kernel size 1×3 or 1×5, stride 1) for feature mapping. Batch normalization layer (BatchNorm), accelerating convergence and stabilizing training. Rectified linear unit activation layer (ReLU), introducing non-linearity. Random deactivation layer (Dropout), with a dropout rate set to 0.1~0.3 to prevent overfitting.

[0029] (3) Multi-scale feature fusion module (Inception): A multi-scale receptive field structure is introduced in the deep layers of the network (e.g., after the 3rd or 4th standard convolutional module). This module contains multiple parallel convolutional paths, each using convolutional kernels of different sizes (1×1, 3×3, 5×5 in parallel) combined with 3×3 max pooling layers. Simultaneously, the input feature map is subjected to multi-scale perception, and then the output feature maps of each path are concatenated along the channel dimension to capture both local details and global contextual information.

[0030] (4) Feature encoding output layer: After multiple convolutional modules, the feature map is compressed into a fixed-dimensional feature vector using a global average pooling (GAP) layer.

[0031] Pre-training is performed through intramodal contrastive learning:

[0032] The self-supervised pre-training strategy employs a contrastive learning-based pre-training scheme. It randomly augments each modality of data through data augmentation (e.g., adding slight noise to acoustic signals, locally masking spectral data, and perturbing frequency points in impedance data), generating two augmented views for the same sample. and As positive sample pairs, data from different samples are used as negative samples. By optimizing the intra-modal InfoNCE loss function, the encoder is driven to extract essential features that are robust to enhancement. The loss function formula is as follows:

[0033] ;

[0034] in, For feature encoder, The cosine similarity function is used. For the i-th original sample, for A positive sample view, Let M be the j-th negative sample, and M represent the number of negative samples. The temperature parameter is typically set to 0.07. After training, the encoder weights are fixed, and the deep features of each modality are output: acoustic features. Spectral characteristics Electrochemical impedance characteristics .

[0035] As a further improvement to the present invention:

[0036] In step (4.1), feature-level fusion and semantic alignment are performed:

[0037] With pre-trained features The input is a vector, and the goal is to achieve cross-modal semantic alignment. The cosine similarity between feature vectors from different modalities is calculated:

[0038] ;

[0039] Pairs of different modal features of the same sample (e.g.) Positive samples are constructed from features of different modalities, and negative samples are constructed from any modal feature pairs. By optimizing the cross-modal InfoNCE loss function, feature representations of the same quality state from different modalities are driven to be closer in semantic space, achieving feature alignment. Finally, the aligned acoustic deep features are obtained. Aligned spectral depth features Deep characteristics of aligned electrochemical impedance .

[0040] The aligned modal features are dynamically weighted and fused through a learnable attention weight network. This network (e.g., two fully connected layers) outputs unnormalized weight scores based on the input aligned features. The attention weights are converted to normalized attention weights using the Softmax function.

[0041] m∈{a,s,e};

[0042] And satisfy .in, Corresponding to , , .

[0043] Finally, a weighted summation is performed to generate a joint feature vector across modalities. :

[0044] ;

[0045] In step (4.2), decision-level fusion:

[0046] Aligned modal features and joint features Inputting each of the corresponding classifiers yields four preliminary predicted probabilities. .

[0047] An optimized weight allocation strategy is employed for the final decision fusion. First, the weights of each classifier are calculated. , , , Independent accuracy on the validation set (i.e. , , , ), to obtain the initial weight proportion :

[0048] ;

[0049] in, Corresponding to , , , .

[0050] Secondly, by analyzing the complementarity between classifiers using the confusion matrix, a subset of samples that are easily misclassified by other classifiers can be extracted. Calculate the target classifier in this subset accuracy and with all classifiers average accuracy In contrast, the complementarity coefficient is introduced and calculated. :

[0051] ;

[0052] Ultimately, the fusion weights Depend on Normalization yields the result, and satisfies... .

[0053] The final quality determination probability is:

[0054] .

[0055] The two-stage fusion mechanism achieves semantic alignment and information complementarity between modalities at the feature level, and enhances the overall reliability of the system through weight optimization at the decision level. It is particularly suitable for quality testing tasks of forest fruit products such as camellia oleifera fruit, which have complex internal structures and diverse external manifestations.

[0056] As a further improvement of the present invention, the single-modal classifier and joint feature classifier Choosing random forests, support vector machines, improved 1D-CNN structures, or lightweight neural networks with fewer layers (such as multilayer perceptrons with ≤3 layers), or different combinations based on modality characteristics (such as support vector machines for single-modality classifiers and improved 1D-CNNs for joint feature classifiers), are all mature models widely used in the industry, with high technical maturity and low maintenance costs. Their hyperparameters are optimized through cross-validation combined with automated search algorithms.

[0057] Other alternative models are as follows:

[0058] Random Forest: 50-200 decision trees, maximum depth 10-30, minimum number of samples per leaf node 2-8. Support Vector Machine: RBF kernel selected, penalty coefficient C∈[0.1,10], amma parameter∈[0.001,0.1]. Improved 1D-CNN structure: 32-128 convolutional kernels, learning rate 0.0001-0.001, 100-300 training epochs.

[0059] The automatic hyperparameter optimization employs 10-fold cross-validation and grid search and cross-validation to automatically adjust hyperparameters without manual intervention, ensuring that the model maintains stable performance in different scenarios.

[0060] As a further improvement of the present invention, the quality label of the camellia fruit in step (5) includes a maturity grade label and / or a mechanical damage degree label.

[0061] To achieve the above-mentioned technical objectives, another technical solution adopted by the present invention is as follows:

[0062] A smart quality detection system for camellia oleifera fruit includes:

[0063] A multimodal data acquisition unit is used to synchronously trigger and receive data from acoustic, spectral, and impedance sensors;

[0064] The computing and control unit includes a preprocessing module for performing step (2) above, a pre-trained feature encoding module for performing step (3) above, a cross-modal feature fusion module for performing step (4) above, and an intelligent decision module for performing step (5) above.

[0065] The human-computer interaction unit is used to configure detection parameters, display detection results, and system status.

[0066] The multimodal data acquisition unit and the computing and control unit are connected via a data bus to achieve real-time data transmission and command synchronization.

[0067] The beneficial effects of this invention are as follows:

[0068] 1. High precision and strong robustness: This invention overcomes the limitations of single-modal detection by complementing multimodal information and deep semantic fusion based on contrastive learning. It has stronger fault tolerance to complex environmental interference and individual sample differences, and has a high detection accuracy.

[0069] 2. This invention systematically solves the core defects pointed out in the background technology (simple feature concatenation, simple voting / weighting in decision-making, and reliance on a large number of annotations) through a coherent technical path of "self-supervised pre-training (laying a solid foundation: using unlabeled data to learn a basic encoder with strong representation capabilities for each modality) → cross-modal contrastive alignment (unifying semantics: mapping features of different modalities to a unified semantic space through contrastive learning) → two-stage adaptive fusion (dynamic complementarity: feature-level dynamic attention + decision-level complementary optimization of weights)", and achieves a leap from shallow information stacking to deep semantic collaboration.

[0070] 3. Low labeling cost and high generalization: The innovative self-supervised pre-training stage utilizes a large amount of unlabeled data to learn general feature representations, significantly reducing the reliance on expensive manually labeled data. The features learned by the model are more universal and easily transferred to different varieties or origins of forest and fruit products.

[0071] 4. Non-destructive and highly efficient throughout the entire process: This invention does not require damaging the sample from data acquisition to analysis and judgment. The single sample detection cycle is short (<30 seconds) and can be directly integrated into the post-production sorting production line to achieve real-time online detection.

[0072] 5. Flexible and scalable architecture: The method framework described in this invention is not limited to camellia fruit. By adjusting the pre-training data and classification labels, it can be easily extended to the detection of other forest fruit products (such as Chinese tallow tree fruit, walnut) or other quality indicators (such as pests and diseases, sugar and acidity). Attached Figure Description

[0073] Figure 1 This is a flowchart illustrating the overall technical route of the method of the present invention.

[0074] Figure 2 This is a schematic diagram of the structure of the multimodal data acquisition system of the present invention.

[0075] Figure 2 Figure (A) in the diagram is a flowchart of the acoustic vibration module, the spectral detection module, and the electrochemical detection module.

[0076] Figure 2 Figure (B) in the diagram is a schematic diagram of the structure of the spectral imaging system.

[0077] Figure 2 Figure (C) in the diagram is a schematic diagram of the vibration signal acquisition system.

[0078] Figure 2Figure (D) in the diagram is a schematic diagram of the electrochemical impedance acquisition system.

[0079] Figure 3 This is a diagram of the feature encoder network architecture for intramodal self-supervised pre-training according to the present invention.

[0080] Figure 4 This is a schematic diagram of the cross-modal contrastive learning and feature fusion of the present invention. Detailed Implementation

[0081] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are for illustrative purposes only and are not intended to limit the scope of protection of this invention.

[0082] like Figure 1 The diagram shown is a flowchart of the overall technical route of the method of the present invention. The operation process of Examples 1-2 is all in accordance with... Figure 1 Execute. (e.g.) Figure 2 As shown, Figure 2 Figure (A) shows the flowchart of the acoustic vibration module, the spectral detection module, and the electrochemical detection module. Data collected by each module is uploaded to the central processing unit for further processing. Figure 2 Figure (B) in the figure is a schematic diagram of the structure of the spectral imaging system, including a spectrometer 1 and a halogen light source 2. Figure 2 Figure (C) in the figure is a schematic diagram of the vibration signal acquisition system, including the laser Doppler vibration meter 3 and the vibratory disk part 4 of the compressed air exciter. Figure 2 Figure (D) in the figure is a schematic diagram of the electrochemical impedance acquisition system, including a fruit sample 5 with electrode plates (custom parallel electrodes) lightly touching its surface, an LCR meter 6, and a laptop computer 7.

[0083] Example 1: Detection of the maturity of camellia fruit.

[0084] 1. Sample and Data Collection:

[0085] During the ripening period, a batch of 200 "Changlin 166" camellia oleifera fruit samples were collected every two weeks, totaling four batches. Based on the forestry industry standard "Technical Specification for Post-Harvest Processing of Camellia Oleifera Fruit" (LY / T 2034-2012) and relevant national standards, combined with the core characterization indicators of camellia oleifera fruit maturity, the samples were divided into four maturity levels: maturity 1 (four weeks before harvest), maturity 2 (two weeks before harvest), maturity 3 (harvest), and maturity 4 (two weeks after harvest). These levels served as objective labels for model training, and the calibration process was non-destructive. After harvesting, all samples were uniformly placed in an environment of 20℃ and 60% relative humidity for 12 hours to equilibrate and eliminate physiological stress and moisture fluctuations during the harvesting process.

[0086] use Figure 2 The data acquisition system collects data synchronously.

[0087] Acoustic vibration modes were tested by using a compressed air exciter (pressure 300 kPa, excitation time 200 ms) to excite fruit vibration. A laser Doppler vibrometer was used to collect the response signal, with a sampling frequency of 10240 Hz and a sampling time of 16 s. Each sample was measured three times. A spectral imaging system was used to collect the 400-1000 nm spectrum with an integration time of 1.4 ms. Dark spectrum and reference spectrum calibration were performed before acquisition. The LCR meter used parallel aluminum electrodes, with a test voltage of 1.0 V and a frequency of 3.98 MHz. It collected eight parameters, including impedance amplitude and phase angle. Each sample was measured three times and the average value was taken.

[0088] 2. Preprocessing and pre-training:

[0089] The acoustic signal is denoised using a 6dB wavelet threshold and the resonant frequency and amplitude features are extracted using autoregressive power spectrum analysis. Input is as follows: Figure 3 The pre-trained model (acoustic modal branch model) shown is used to obtain deep acoustic features. .

[0090] After preprocessing the spectral data using Standard Normal Transform (SNV), 15 characteristic wavelengths were selected using the Continuous Projection (SPA) algorithm. Input is as follows: Figure 3 The pre-trained model (spectral modal branch model) shown is used to obtain deep spectral features. .

[0091] After logarithmic transformation and normalization, the electrochemical impedance data were used to extract four core impedance features via partial least squares (PLS). Input is as follows: Figure 3 The pre-trained model (electrochemical modal branching model) shown is used to obtain deep electrical features. .

[0092] like Figure 3 As described above, three independent one-dimensional convolutional neural network branches are constructed as feature encoders, each processing a different modality of data (such as acoustic vibration signals, spectral data sequences, or electrochemical impedance parameter sequences). Each branch serves as an independent feature encoder. Each branch employs a deep stacked, improved 1D-CNN design, with the following modular structure:

[0093] (1) Input layer: Receives preprocessed modal timing data;

[0094] (2) Convolutional Layers: Composed of multiple standard convolutional modules. One-dimensional convolutional layer (Conv1D), using multiple convolutional kernels (kernel size 1×3 or 1×5, stride 1) for feature mapping. Batch normalization layer (BatchNorm), accelerating convergence and stabilizing training. Modified linear unit activation layer (ReLU), introducing non-linearity. Random deactivation layer (Dropout), with a dropout rate set to 0.1~0.3 to prevent overfitting.

[0095] (3) Multi-scale feature fusion module (Inception): A multi-scale receptive field structure is introduced in the deep layers of the network (e.g., after the 3rd or 4th standard convolutional module). This module contains multiple parallel convolutional paths, each using convolutional kernels of different sizes (1×1, 3×3, 5×5 in parallel) combined with 3×3 max pooling layers. Simultaneously, the input feature map is subjected to multi-scale perception, and then the output feature maps of each path are concatenated along the channel dimension to capture both local details and global contextual information.

[0096] (4) Feature encoding output layer: After multiple convolutional modules, the feature map is compressed into a fixed-dimensional feature vector using a global average pooling (GAP) layer.

[0097] 3. Cross-modal fusion, such as Figure 4 As shown:

[0098] calculate and , and , and Cosine similarity is used to construct cross-modal positive and negative sample pairs, and feature semantic alignment is achieved through cross-modal InfoNCE loss optimization, resulting in aligned features. .

[0099] Automatically calculate attention weights using an attention network. =0.45、 =0.35、 =0.2, generate ,in, .

[0100] Four classifiers were built (all using random forest), and the hyperparameters were optimized using 10-fold cross-validation. The number of decision trees was 150, the maximum depth was 25, and the minimum number of samples per leaf node was 4.

[0101] The following example demonstrates the input, calculation, and output of a specific sample of data:

[0102] Classifier ,enter Output The independent accuracy on the validation set was 92.5%. The percentage of accuracy was 0.274.

[0103] Classifier ,enter Output The independent accuracy on the validation set was 88.0%. The percentage of accuracy was 0.251.

[0104] Classifier ,enter Output The independent accuracy on the validation set was 75.0%. The percentage of accuracy was 0.214.

[0105] Classifier ,enter Output The independent accuracy on the validation set was 95.0%. The percentage of accuracy was 0.271.

[0106] Four random forest classifiers output predicted probabilities, respectively. On the validation set, the accuracies of the four classifiers are 92.5%, 88.0%, 75.0%, and 95.0%, respectively.

[0107] Through confusion matrix analysis and grid search, , , , Correction coefficients of 1.82, 1.0, 0.47, and 0.55 were assigned respectively, and the fusion weights were obtained by multiplying them by the accuracy percentage. =0.5、 =0.25、 =0.1、 =0.15.

[0108] Finally, input the four probabilities into the formula. ,get =[0.02,0.04,0.91,0.03], take the maximum probability value of maturity 3 as the final prediction result.

[0109] 4. Results: The system achieved an overall accuracy of 96.2% in maturity grading of the test set, which is significantly higher than any single-modal classifier.

[0110] Example 2: Detection of damage to camellia fruit.

[0111] 1. Sample and data collection: 200 samples each of no damage, mild damage, moderate damage and severe damage were prepared through drop tests.

[0112] Eighty hundred fresh camellia fruit samples were selected, and a free-fall test was conducted to simulate impact damage during post-harvest transportation and processing. Four damage levels were set: control group (no damage), minor damage (0.6m, simulating sorting and conveyor collision), moderate damage (1.2m, simulating loading and unloading tipping impact), and severe damage (1.8m, simulating falling from a low branch during harvesting), with 200 samples in each group. Sand was laid on the impact surface to avoid multiple impacts or shell breakage of the samples.

[0113] A compressed air excitation unit (pressure 300 kPa, excitation time 200 ms) was used to excite fruit vibration. A laser Doppler vibrometer was used to collect the response signal. The sampling frequency was 10240 Hz and the sampling time was 16 s. Each sample was measured three times to extract dynamic characteristics such as resonant frequency shift and damping coefficient. For spectral imaging, a self-built hyperspectral system (wavelength range 400-1000 nm) was used to collect the spectrum of the region near the impact point. The average spectrum of the region of interest (ROI) of the sample was extracted using ENVI software, and 15 characteristic wavelengths related to cell structure damage and water distribution changes were screened using the continuous projection algorithm (SPA). Using an LCR meter and a custom parallel electrode, eight electrical parameters, including impedance amplitude, phase angle, and parallel equivalent resistance, were collected at a test voltage of 1.0 V and a frequency of 3.98 MHz. Each sample was measured three times and the average value was taken to improve data stability.

[0114] 2. Preprocessing and pre-training:

[0115] Acoustic signals are denoised using db6 wavelet thresholding, and time-frequency domain features are extracted using an autoregressive model. These features are then input into a pre-trained model to obtain deep acoustic features. .

[0116] Spectral data underwent standard normal variable transformation (SNV) to eliminate scattering interference, and then 15 damage-sensitive bands were extracted using SPA. These were then input into a pre-trained model to obtain deep spectral features. .

[0117] After logarithmic transformation and standardization, electrochemical impedance data were used to screen out four core features related to cell membrane integrity and ion permeability using partial least squares (PLS). These features were then input into a pre-trained model to obtain deeper electrical features. .

[0118] 3. Cross-modal fusion:

[0119] Calculate the cosine similarity of cross-modal features, construct cross-modal positive and negative sample pairs, and achieve semantic alignment through cross-modal InfoNCE loss optimization to obtain the aligned features. .

[0120] Automatically calculate attention weights using an attention network. =0.4、 =0.3、 =0.3, weighted summation yields cross-modal joint features .

[0121] Four classifiers (with differentiated combinations) were built, and the hyperparameters were optimized using 10-fold cross-validation.

[0122] The following example demonstrates the input, calculation, and output of a specific sample of data:

[0123] Classifier (Support Vector Machine), kernel function chosen: RBF kernel, penalty coefficient C=5, gamma parameter=0.01. Input Output The independent accuracy on the validation set was 92.5%. The percentage of accuracy was 0.261%.

[0124] Classifier (Support Vector Machine), kernel function chosen: RBF kernel, penalty coefficient C=6, gamma parameter=0.008. Input Output The independent accuracy on the validation set was 90.0%. The percentage of accuracy was 0.253.

[0125] Classifier (Support Vector Machine), kernel function chosen: RBF kernel, penalty coefficient C=4, gamma parameter=0.012. Input Output The independent accuracy on the validation set was 78.0%. The percentage of accuracy was 0.219.

[0126] Classifier (Improved 1D-CNN), 64 convolutional kernels, learning rate 0.0006, 200 training epochs, Dropout rate 0.2. Input Output The independent accuracy on the validation set was 95.0%. The percentage of accuracy was 0.267.

[0127] Through confusion matrix analysis and Bayesian optimization iteration screening, the following were selected: , , , Correction coefficients of 1.91, 1.19, 0.46, and 0.37 were assigned respectively, and the fusion weights were obtained by multiplying them by the accuracy percentage. =0.5、 =0.3、 =0.1、 =0.1.

[0128] Finally, input the four probabilities into the formula. ,get =[0.92,0.05,0.02,0.01], taking the maximum probability of no damage as the final prediction result.

[0129] 4. Results: The detection rate of damage (especially minor damage) of this method reached 94.8%, while that of single spectral method was only 81.5%, which proves the key role of multimodal fusion in improving the sensitivity of damage detection.

[0130] Industrial applicability:

[0131] The effectiveness of the method and system described in this invention has been verified through the above embodiments. The system hardware is integrated with conventional industrial sensors, and the algorithm can be deployed on embedded industrial control computers or industrial servers. It can seamlessly integrate with existing post-harvest cleaning, sorting, and packaging production lines for camellia oleifera fruits, achieving fully automated online quality detection and grading. It has clear industrial application prospects and promotional value.

[0132] Beneficial effects:

[0133] 1. The entire process is non-contact / micro-contact and non-destructive, ensuring no impact on subsequent fruit processing, storage, and sales, thus reducing losses. Data acquisition and processing are fully automated and can be operated by a single person. It utilizes conventional LCR meters and laser Doppler vibration meters, eliminating the need for expensive specialized testing instruments. The highly automated data acquisition and processing workflow requires no manual intervention from professional technicians, lowering the operational threshold and facilitating widespread application in grassroots production areas and processing enterprises, significantly reducing industry costs.

[0134] 2. The entire testing process takes less than 30 seconds per sample, supports large-scale online testing, and the testing system can be directly integrated into existing post-production sorting production lines. The multimodal fusion and comparative learning mechanism can resist interference from production environments such as temperature, humidity, and vibration, exhibiting strong anti-interference capabilities and high accuracy.

[0135] 3. It is applicable to a variety of forest and fruit products such as camellia fruit, Chinese privet fruit, walnut, and chestnut. It can be applied to the entire chain of scenarios such as rapid grading at the harvest site, random inspection during storage, and cold chain logistics node detection. It does not require changing equipment or models and has strong adaptability.

[0136] 4. This invention solves many of the shortcomings of traditional detection methods by combining deep fusion of multimodal data with advanced machine learning algorithms, and provides a new solution for large-scale and intelligent detection of the quality of forest and fruit products. It has important industrial application value and promotion prospects.

[0137] The scope of protection of this invention is not limited to the above embodiments. Any quality detection of forest and fruit products based on the method of this invention, achieved by modifying modal parameters (such as adjusting the acquisition frequency and wavelength range), adjusting the model structure (such as increasing or decreasing the number of convolutional layers and changing the optimizer), expanding the detection objects (such as other forest and fruit products), or the detection indicators (such as pests and diseases, sugar content), etc., shall fall within the scope of protection of this invention.

Claims

1. A method for intelligent detection of camellia fruit quality based on multimodal fusion and self-supervised pre-training, characterized in that, Includes the following steps: (1) Synchronous acquisition of multimodal data: The acoustic vibration response signal, spectral data in the visible to near-infrared band and electrochemical impedance spectral parameters of the target camellia fruit sample are acquired simultaneously through the non-contact acoustic vibration detection module, spectral detection module and electrochemical impedance detection module. (2) Cross-modal data preprocessing: Modal-specific preprocessing is performed on the original modal data collected in step (1); wavelet denoising and power spectrum feature extraction are performed on the acoustic vibration signal to obtain the basic acoustic features; Scattering correction, smoothing, and characteristic wavelength selection are performed on spectral data to obtain basic spectral characteristics; The electrochemical impedance data are transformed and reduced in dimensionality to obtain the basic impedance characteristics. (3) Intramodal self-supervised pre-training feature learning: Construct three independent one-dimensional convolutional neural network branches as feature encoders, respectively processing acoustic, spectral and impedance basic features; Random augmentation is performed on the data of each modality to generate different augmented views of the same sample within the modality as positive sample pairs, and data from different samples as negative samples. By optimizing the intramodal contrastive learning loss function, the encoders of each branch are trained to extract deep feature representations that are robust to augmentation and have discriminative power. After training, with encoder parameters fixed, the deep acoustic features are extracted and output. Spectral depth features and deep characteristics of electrochemical impedance ; (4) Cross-modal feature fusion: (4.1) Cross-modal semantic alignment and feature-level fusion: based on the results obtained in step (3) , , Using this as input, the cosine similarity between feature vectors of different modalities is calculated. Feature vectors of different modalities from the same sample are constructed as positive sample pairs, and feature vectors of any modality from different samples are constructed as negative sample pairs. By optimizing the cross-modal contrastive learning loss function, the similarity of features in positive sample pairs in the semantic space is maximized, and the similarity of features in negative sample pairs is minimized. This achieves the alignment of acoustic, spectral, and impedance features in the common semantic space, yielding the aligned deep acoustic features. Aligned spectral depth features Deep characteristics of aligned electrochemical impedance ; The aligned modal features are dynamically weighted and summed through a learnable attention weight network to generate a unified cross-modal joint feature vector. ;in, ; ; Attention weight The attention weight network outputs unnormalized weight scores based on the alignment features of the input. The result obtained after transformation using the Softmax function is: ； and , Corresponding to , , ; (4.2) Decision-level fusion: aligning the deep acoustic features Input to a single-modal classifier Single-modal classifier Output quality prediction probability distribution ;aligned spectral depth features Input to a single-modal classifier Single-modal classifier Output quality prediction probability distribution ; to align the deep features of electrochemical impedance Input to a single-modal classifier Single-modal classifier Output quality prediction probability distribution ;Joint features across modalities Input to joint feature classifier Joint feature classifier Output quality prediction probability distribution Based on the accuracy of each classifier on the independent validation set and the complementarity of their prediction results, the final fusion weights are determined through a weight allocation strategy. ,in The final probability of judging the quality of the camellia fruit is calculated by the following formula: ; (5) Model training and deployment: Using a sample dataset labeled with the quality label of camellia fruit, supervised learning is used to jointly train and optimize all classifiers, attention weight networks and fusion weights mentioned in step (4) to determine their final parameters, thereby obtaining a complete detection model that can be used for quality judgment; the trained model is deployed to the detection system to achieve automated and non-destructive intelligent quality detection of new camellia fruit samples.

2. The intelligent detection method for camellia fruit quality based on multimodal fusion and self-supervised pre-training as described in claim 1, characterized in that, In step (1), the acoustic vibration detection module includes a compressed air excitation unit and a laser Doppler vibration meter. The excitation pressure is 300 kPa, the excitation time is 200 ms, and the vibration signal acquisition frequency range is 50 Hz-2000 Hz.

3. The intelligent detection method for camellia fruit quality based on multimodal fusion and self-supervised pre-training as described in claim 1, characterized in that, In step (1), the wavelength acquisition range of the spectral detection module is 400nm-1000nm, and the parameter acquisition frequency range of the electrochemical impedance detection module is 100Hz-5MHz.

4. The intelligent detection method for camellia fruit quality based on multimodal fusion and self-supervised pre-training as described in claim 1, characterized in that, The intra-modal contrastive learning loss function in step (3) is the InfoNCE loss function. ； in, For feature encoder, The cosine similarity function is used. For the i-th original sample, for A positive sample view, Let M be the j-th negative sample, M represent the number of negative samples, and the temperature parameter τ has a value range of 0.07; the deep layers of the one-dimensional convolutional neural network branch contain the Inception structure with parallel multi-scale convolutional kernels.

5. The intelligent detection method for camellia fruit quality based on multimodal fusion and self-supervised pre-training as described in claim 1, characterized in that, The cross-modal contrastive learning loss function in step (4.1) is the InfoNCE loss function based on cosine similarity.

6. The intelligent detection method for camellia fruit quality based on multimodal fusion and self-supervised pre-training as described in claim 1, characterized in that, The weight allocation strategy in step (4.2) specifically includes: firstly calculating the initial proportion of the independent accuracy of each classifier to the sum of the four. Secondly, confusion matrix analysis is used to identify sample subsets that other classifiers are prone to misclassification, and the accuracy of the target classifier on these subsets is calculated. Based on this, the complementarity coefficient is calculated. The final weights are determined by The result is obtained by normalization.

7. The intelligent detection method for camellia fruit quality based on multimodal fusion and self-supervised pre-training as described in claim 1, characterized in that, The single-modal classifier and joint feature classifier Random forest, support vector machine, or improved 1D-CNN were selected for all studies; their hyperparameters were optimized through cross-validation combined with an automated search algorithm.

8. The intelligent detection method for camellia fruit quality based on multimodal fusion and self-supervised pre-training as described in claim 1, characterized in that, The quality label for the camellia fruit in step (5) includes a maturity grade label and / or a mechanical damage degree label.

9. An intelligent quality detection system for camellia fruit, used to implement the method according to any one of claims 1-8, characterized in that, The system includes: A multimodal data acquisition unit is used to synchronously trigger and receive data from acoustic, spectral, and impedance sensors; The computing and control unit includes a preprocessing module, a pre-trained feature encoding module, a cross-modal feature fusion module, and an intelligent decision module, which are used to execute all the algorithm processes from step (2) to step (5) in claim 1 respectively; The human-computer interaction unit is used to configure detection parameters, display detection results, and system status. The multimodal data acquisition unit and the computing and control unit are connected via a data bus to achieve real-time data transmission and command synchronization.

Images