A tomato maturity intelligent recognition system and method based on olfactory and visual bimodal fusion

Abstract

The application relates to the field of agricultural equipment and discloses a tomato maturity intelligent recognition system and method based on olfaction and vision bimodal fusion. The system comprises an image acquisition module 1, a gas acquisition module 2, a gas distribution visualization module 3, a background tomato depth separation module 4, an RGB image and gas modal fusion feature network 5 and a maturity detection output display 6. The method comprises the following steps: synchronously acquiring a visual image and surrounding gas of tomatoes; respectively extracting visual and olfactory features; fusing the bimodal features; and recognizing a maturity grade based on the fused features by using a classification model. The application overcomes the shortcomings that a single visual mode is observed from a single side, is easily deceived by color and is interfered by the environment by fusing appearance and gas information on the surface of the fruit, realizes more accurate and robust intelligent recognition of the maturity of the tomatoes and is suitable for agricultural automatic picking and sorting.

Description

Technical Field

[0001] This invention relates to the field of agricultural equipment, and in particular to an intelligent recognition system and method for tomato ripeness based on the fusion of olfactory and visual dual modalities. Background Technology

[0002] As a widely grown and consumed fruit and vegetable globally, the ripeness of tomatoes directly determines their taste, nutritional value, shelf life, and market value. Accurately identifying tomato ripeness is crucial for precise harvesting, grading, packaging, cold chain transportation, and market circulation, and is of great significance for improving the economic benefits and market competitiveness of the tomato industry.

[0003] Currently, tomato maturity identification methods are mainly divided into two categories: manual identification and single-modal machine identification. Manual identification relies on the operator's experience and judgment, identifying tomatoes by observing their color, size, and smell. However, this method has significant drawbacks: firstly, it is highly subjective, with different operators having different judgment standards, leading to unstable identification accuracy; secondly, it is inefficient, failing to meet the needs of large-scale industrial production, and prolonged work can easily lead to operator fatigue, further reducing identification accuracy. Single-modal machine identification methods mainly include visual identification and olfactory identification. Visual identification methods collect image information of tomatoes, extract features such as color, texture, and shape, and use machine learning or deep learning models to classify maturity. This method has the advantages of being non-contact and highly efficient, but it is greatly affected by environmental factors, such as light intensity and shooting angle, which can lead to inaccurate image feature extraction, thus affecting identification accuracy; at the same time, for tomato varieties with indistinct color differences or tomatoes in the transitional ripening stage, it is difficult to achieve accurate identification relying solely on visual features.

[0004] Olfactory recognition methods use gas sensor arrays to collect volatile gas components (such as ethylene and acetaldehyde) released by tomatoes. The types and concentrations of these gas components change as the tomatoes ripen, and ripeness is identified by analyzing these gas characteristics. However, single-sense olfactory recognition methods also have shortcomings: gas sensors are easily interfered with by other volatile gases in the environment, leading to errors in feature extraction; and different tomato varieties release different volatile gas components, resulting in poor model versatility and requiring retraining for different varieties, increasing application costs.

[0005] To address the issues of low accuracy, poor anti-interference ability, and insufficient versatility of single-modal recognition methods, this invention proposes an intelligent tomato ripeness recognition system and method based on the fusion of olfactory and visual dual modalities. By combining the advantages of both modalities, it achieves accurate and efficient recognition of tomato ripeness. Summary of the Invention

[0006] This invention belongs to the field of agricultural intelligent detection and equipment technology. It addresses two major bottlenecks in tomato maturity identification in real-world production environments. First, single-modal methods relying on visual images experience a significant increase in false detection rates when light levels change, shooting angles change, or backgrounds become complex, making it difficult to reliably distinguish samples in the ripening transition stage. Second, single-modal methods relying on electronic olfaction suffer from insufficient stability in the presence of stray gases and differences in volatile components between varieties, exhibiting limited cross-variety generalization ability. This invention proposes an intelligent identification scheme that deeply integrates olfactory and visual information, employing cross-modal spatiotemporal alignment and network-level fusion strategies to improve accuracy, robustness, and real-time performance in sorting lines and post-harvest grading processes.

[0007] The system of this invention consists of six functional units: an image acquisition module, a gas acquisition module, a gas distribution visualization module, a background tomato depth separation module, an RGB image and gas modality fusion feature network, and a maturity detection and output display unit. These six modules work in a synchronized, time-coordinated manner on the same embedded computing platform, forming a top-down closed-loop process. The process sequence is: synchronous acquisition, collaborative preprocessing, multi-scale fusion, online judgment, result display, and data reporting. The system interface, clock source, and cache management adopt a unified standard to ensure precise alignment of cross-modal data within a millisecond timescale.

[0008] The device employs an integrated, sealed detection chamber. The chamber is made of transparent acrylic, with an anti-reflective black film covering the inner wall. An Intel D304 depth camera and a ring-shaped constant-illuminance supplementary light are fixed at the top, with the vertical distance from the lens to the sample fixed at 40 cm. The viewing angle and focal length remain locked after calibration. An electronic olfactory array consisting of eight metal oxide sensors is installed on the sidewalls of the chamber, with independent channels for ethylene, acetaldehyde, ethanol, and other mature volatile compounds. Temperature and humidity sensors are installed at the bottom of the chamber to perform environmental compensation. The main control unit is an embedded industrial computer responsible for model inference, cross-modal fusion, and human-computer interaction; a microcontroller handles camera triggering, sensor sampling, and illumination control; a touchscreen displays detection results, task status, and historical records, and provides batch export and network reporting functions.

[0009] The gas distribution visualization module maps the instantaneous concentration vectors of 8 channels into a fixed-size 32×32 pseudo-color heatmap. The mapping steps include linear normalization, channel rearrangement, bilinear interpolation, and histogram equalization. This heatmap serves as the spatial representation input to the downstream network for the olfactory side, enabling the convolutional operator to process olfactory features. The background tomato depth separation module uses the distance map output by the depth camera as a basis, setting a single-threshold distance threshold of 0.35 m and a morphological infill scale of 0.1 m. In each frame, it retains only the largest connected components within the target depth range, removes distant background and near-field clutter, and outputs a binary mask to constrain the effective area of ​​the visual branches.

[0010] Data acquisition and time alignment employ a strict synchronization strategy. The visual side acquires RGB frames and depth frames at a resolution of 640×640, with a frame rate of 30 frames per second. The olfactory side acquires 8-channel temporal signals at a sampling rate of 100 Hz, along with temperature and humidity records. A unified timestamp is generated and distributed by the main control unit. The synchronization submodule aggregates each frame of the RGB image with olfactory data from the preceding and following seconds, using a sliding window averaging and first-order polynomial interpolation to generate a 3-channel olfactory tensor perfectly aligned with the image. Visual side preprocessing includes proportional scaling, center cropping, geometric normalization, brightness and contrast enhancement, color temperature and saturation perturbation, and mean-variance normalization. Olfactory side preprocessing includes temperature and humidity compensation, third-order Butterworth low-pass filtering, drift correction, Z-score normalization, and outlier truncation. All preprocessing uses the same work cycle between the two modalities to ensure temporal and statistical consistency.

[0011] The fusion feature network employs a dual-branch isomorphic structure. The visual branch calls upon the backbone and neck structure of YOLO v11 to output feature maps at three scales: P3, P4, and P5, and loads pre-trained visual weights to accelerate convergence. The olfactory branch uses the same convolutional, CSP, and FPN structure as the visual branch, receiving a 3-channel olfactory tensor at the input. Multi-scale fusion performs shape alignment and channel alignment at each scale, concatenating features of the same scale along the channel dimension. Subsequently, 1×1 convolutions, batch normalization, and SiLU activation are used to complete channel recalibration and compression. Then, SE channel attention and lightweight spatial attention are concatenated to increase the proportion of effective features and suppress noise response. The fused three-scale features are fed into the YOLOv11 detector head for classification and regression. In the post-processing stage, non-maximum suppression is used to output maturity level and confidence, and spatial consistency is checked based on the depth mask to reduce edge misjudgment.

[0012] The training strategy employs end-to-end joint optimization. Training data consists of synchronously acquired image, depth, and olfactory triples, with a batch size of 16, 300 training epochs, and a base learning rate of 0.01. Cosine annealing scheduling and a 5-epoch warm-up are used. The weight decay coefficient is set to 0.0005, the gradient clipping threshold is set to 1.0, and exponential moving average is introduced to improve stability and evaluation consistency. Geometric augmentation is fully synchronized between the visual and olfactory branches, while color augmentation applies only to the visual branch. The loss function is a weighted sum of classification cross-entropy and CIOU regression loss, with a weight ratio of 1:1. The validation phase uses the same preprocessing, alignment, and fusion rules as training. Evaluation metrics include accuracy, recall, mean precision, inference latency, and throughput.

[0013] The online inference process consists of five steps. Step 1: The acquisition module starts and generates a synchronization timestamp. Step 2: The depth separation module retains the tomato region within the target depth range and outputs a binary mask. Step 3: The olfactory side updates the 1-second sliding buffer and completes compensation, filtering, and normalization, while generating a 3-channel olfactory tensor aligned with the current image frame. Step 4: The dual-branch network completes feature extraction and multi-scale fusion, outputting the category and location at the detection head. Step 5: Non-maximum suppression yields the final maturity level and confidence score. The results are simultaneously displayed on the touchscreen using text and color labels, and the detection records are written to the local database and reported via a wireless network. The single-machine processing capacity reaches 10 samples per second, with end-to-end latency controlled within 100ms.

[0014] The methodology operates on a fixed timeline. The operator places the sample in the center of the tray and allows it to stand for 60 seconds to stabilize and evaporate before initiating a detection task. The system completes the determination according to the online inference process, archiving the maturity level, confidence level, corresponding image, olfactory heatmap, and key intermediate variables. At the end of each day, offline incremental training is performed to update the model parameters, using data from newly added and misclassified samples to maintain long-term accuracy.

[0015] Key parameters and boundary conditions are uniformly set. The depth threshold is 0.35 m, the morphological imputation scale is 0.1 m, the RGB resolution is 640×640, the visual frame rate is 30 frames per second, the olfactory sampling rate is 100 Hz, the temporal aggregation window is 1 s, the olfactory pseudo-image size is 32×32, and the 1×1 convolution output channel at the fusion point is 50% of the original stitched channel number. The training batch size is 16, the number of training epochs is 300, the base learning rate is 0.01, the weight decay is 0.0005, the gradient clipping threshold is 1.0, and the loss weighting ratio is 1:1. The above parameters are fixed as default configurations in the delivered version and are provided as read-only displays and audit logs in the maintenance interface.

[0016] In the effectiveness verification phase, a synchronous sample library was constructed using five tomato varieties and four maturity levels. Data were collected at different time periods, under varying light intensities, and in different ambient temperatures and humidity conditions. Cross-validation was used to provide statistical results. The average accuracy of the dual-modal fusion model was significantly higher than that of the single visual and single olfactory models. The model demonstrated significantly enhanced stability in identifying samples during the ripening transition phase, maintaining high accuracy and recall even under conditions of low light, occlusion, and methanol contamination. Continuous production line testing showed that the overall throughput reached the level of multiple samples per second, meeting the real-time requirements of batch online grading.

[0017] Compared with the prior art, the present invention has three significant beneficial effects.

[0018] 1. It provides three fixed paths: end-to-end fusion, decision-level fusion, and feature-level fusion, covering three types of engineering scenarios: high precision, speed, and generality, with clear boundaries for deployment and switching.

[0019] 2. By leveraging the complementary advantages of vision and smell, the ability to distinguish between samples with similar colors and subtle morphological differences is enhanced, significantly improving overall accuracy and reducing the error rate during the maturation transition stage.

[0020] 3. Relying on deep separation, environmental compensation, and attention enhancement, it maintains stable output under complex conditions such as low light, textureless surfaces, occlusion, and stray gas interference. It completes automated online processing from acquisition to identification at a fixed pace, and has the ability to be deployed on a large scale and is traceable.

[0021] The system structure, algorithm flow, and key parameters provided in this invention are all fixed settings. Each module can independently complete its intended function and maintain coordination with a unified clock through a fixed interface when the system is cascaded. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating a tomato ripeness intelligent recognition system and method based on the fusion of olfactory and visual modalities. Figure 2 This is a schematic diagram of the image acquisition module; Figure 3 This is a schematic diagram of the flow guiding structure of the gas sensor array; Figure 4 This is a front view of the gas sensor array flow guide structure; Figure 5 This is a flowchart illustrating the visual processing workflow. Figure 6 This is a schematic diagram of the olfactory processing flow. Detailed Implementation

[0023] This invention proposes an intelligent tomato ripeness recognition system and method based on the fusion of olfactory and visual modalities. The invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0024] like Figure 1 The embodiment of the present invention shown includes the following main working devices: image acquisition module 1, gas acquisition module 2, gas distribution visualization module 3, background tomato depth separation module 4, RGB image and gas mode fusion feature network 5, and maturity detection output display 6. Each system works together to complete the task.

[0025] (a) Hardware deployment 1. Detection chamber: made of transparent acrylic material, with a black light-absorbing film on the inner wall, a camera and supplementary light installed at the top, a sensor array fixed on the side, and a sample tray at the bottom; 2. Main control unit: NVIDIA Jetson Nano (4GB memory) runs the fusion algorithm and recognition model, and STM32F103 controls the peripherals; 3. Interface configuration: INTEL D304 (camera), I2C (sensor), UART (WiFi / display), HDMI (touchscreen).

[0026] (II) Software Implementation 1. Development environment: Python 3.8, OpenCV 4.5, TensorFlow 2.5; 2. Core code snippet: Feature layer fusion: joint_feature = np.hstack((visual_feature, olfactory_feature)) Decision-making level fusion: weighted_result = 0.89*svm_result + 0.86*rf_result End-to-end network: constructed using PYTORCH, with cross-attention layers achieving feature interaction through matrix multiplication.

[0027] (III) Experimental Verification 1. Experimental sample: 200 samples of each of the 5 tomato varieties (4 grades × 50 samples), for a total of 1000 samples; 2. Test conditions: normal light (5000 lux), strong light (6500 lux), weak light (3500 lux), environment containing 10 ppm methanol impurity gas; 3. Evaluation metrics: accuracy, processing speed, and interference resistance retention rate; 4. Results Analysis: All three fusion strategies are significantly better than single-modal fusion. End-to-end fusion is the best in terms of accuracy and anti-interference, while decision-level fusion is the best in terms of speed.

[0028] This embodiment is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A tomato ripeness intelligent recognition system based on olfactory and visual dual-modal fusion, characterized in that, include: Image acquisition module 1 uses an RGB-D depth camera to synchronously acquire RGB images and depth maps of the tomatoes to be detected at a resolution of 640×640 and 30 frames per second. Gas acquisition module 2 uses a gas sensor array consisting of 8 metal oxide sensors to acquire gas samples from the space around the tomato under test at a sampling rate of 100 Hz, obtain 8-channel instantaneous concentration signals, and simultaneously record temperature and humidity. The gas distribution visualization module 3 is connected to the gas acquisition module 2 and is used to map the 8-channel instantaneous concentration signal into a fixed-size 32×32 pseudo-color heat map. The mapping steps are linear normalization, channel rearrangement, bilinear interpolation and histogram equalization. This pseudo-color heat map is used as a spatialized representation of the olfactory side and input into the downstream convolutional network, so that the convolutional operator can process olfactory features in a way consistent with the processing of visual feature maps. Background tomato depth separation module 4 is used to extract the target region and separate the background of the acquired tomato image. A distance threshold of 0.35 m is set on the depth map and a morphological filling scale of 0.1 m is used to retain only the largest connected component within the target depth range to form a target binary mask and remove irrelevant interference information. The RGB image and gas modality fusion feature network 5 adopts a dual-branch isomorphic structure with a visual branch and an olfactory branch. The visual branch calls the backbone and neck of YOLO v11 to output visual feature maps at three scales: P3, P4, and P5. The olfactory branch adopts the same convolution, CSP, and FPN structure as the visual branch, and takes the 3-channel olfactory tensor formed by averaging the pseudo-color heatmap output by the gas distribution visualization module 3 through a sliding window as input. At each scale of P3, P4, and P5, the visual feature map and the olfactory feature map are aligned in shape and channel, and then stitched along the channel dimension. Then, they are sequentially subjected to 1×1 convolution, batch normalization, and SiLU activation to complete channel recalibration and compression. SE channel attention is superimposed to enhance effective features and suppress noise response. The RGB image and gas modality fusion feature network 5 is connected to the background tomato depth separation module 4, so that the fusion calculation is limited to the effective spatial area constrained by the target binary mask. The maturity detection output display module 6 is connected to the RGB image and gas mode fusion feature network 5. It is used to send the fused multi-scale features into the YOLO v11 detection head to complete classification and regression. After non-maximum suppression, it outputs the maturity level and confidence level. After using the target binary mask to perform spatial consistency verification on the detection results, it outputs the visualized detection results.

2. The system according to claim 1, characterized in that, The RGB image and gas modality fusion feature network 5 further sets up a cross-attention layer between the visual branch and the olfactory branch. The cross-attention layer realizes cross-modal feature interaction through matrix multiplication and completes end-to-end fusion of visual features and olfactory features in the middle layer of the network.

3. A method for recognizing tomato ripeness based on olfactory and visual dual-modal fusion, characterized in that, Includes the following steps: Step A: Simultaneously acquire RGB images and depth maps of the tomato under test at a resolution of 640×640 and 30 frames per second, and acquire gas samples of the space around the tomato under test at a sampling rate of 100 Hz, obtain 8-channel instantaneous concentration signals, and simultaneously record temperature and humidity. Step B: Map the 8-channel instantaneous concentration signal into a fixed-size 32×32 pseudo-color heatmap. The mapping steps include linear normalization, channel rearrangement, bilinear interpolation, and histogram equalization. The pseudo-color heatmap is then used as a spatialized representation of the olfactory side and input into the downstream convolutional network. Step C: Extract the target region and separate the background from the depth map. Set a distance threshold of 0.35 m on the depth map and use a morphological filling scale of 0.1 m. Only retain the largest connected region within the target depth range to form a target binary mask and remove distant background and near-field clutter. Step D: Feature extraction and fusion are performed using a dual-branch isomorphic structure of visual and olfactory branches. The visual branch calls the backbone and neck of YOLO v11 to output visual feature maps at three scales: P3, P4, and P5. The olfactory branch adopts the same convolution, CSP, and FPN structure as the visual branch, and takes the 3-channel olfactory tensor formed by averaging the pseudo-color heatmap through a sliding window as input. Step E: At each scale of P3, P4, and P5, shape alignment and channel alignment are performed on the visual feature map and the olfactory feature map, and then they are stitched along the channel dimension. Then, 1×1 convolution, batch normalization and SiLU activation are performed to complete channel recalibration and compression. SE channel attention is superimposed to enhance effective features and suppress noise response, while limiting the fusion calculation to the effective spatial area constrained by the target binary mask. Step F: The fused multi-scale features are fed into the YOLO v11 detection head to complete classification and regression. After non-maximum suppression, the maturity level and confidence level are output. The target binary mask is used to perform spatial consistency verification on the detection results and then the visualized detection results are output.

4. The method according to claim 3, characterized in that, The RGB images and depth maps were acquired using an Intel D304 depth camera, along with a ring-shaped constant brightness fill light. The distance from the lens to the sample tray was fixed at 40 cm, and the intrinsic and extrinsic parameters were locked after a one-time factory calibration.

5. The method according to claim 3, characterized in that, When performing the feature extraction and fusion, a lightweight spatial attention module is further superimposed after the SE channel attention is superimposed; when performing the spatial consistency check, each detection box output by the non-maximum suppression is compared pixel by pixel with the target binary mask, eliminating false edge detections that are spatially inconsistent with the target binary mask, and retaining the detection results that pass the spatial consistency check.

6. The method according to claim 3, characterized in that, Visual preprocessing includes proportional scaling, center cropping, geometric normalization, brightness and contrast enhancement, color temperature and saturation perturbation, and mean and variance normalization; olfactory preprocessing includes temperature and humidity compensation, third-order Butterworth low-pass filtering, drift correction, Z-score normalization, and outlier truncation.

7. The method according to any one of claims 3 to 6, characterized in that, The online inference single-machine processing capacity reaches 10 samples per second, with an end-to-end latency of no more than 100 ms. The output includes text labels and color labels, and is archived together with RGB images, target binary masks, maturity levels, and timestamps.

Images