A Non-destructive Testing System for Apple Ripeness Based on Multimodal Perception Fusion

Abstract

This invention belongs to the field of agricultural automation and computer vision technology, and relates to a non-destructive testing system for apple ripeness based on multimodal perception fusion. The system includes a camera, a spectral sensor, a main control unit, a visual detection model, and a spectral perception model. The camera, spectral sensor, and main control unit are connected. The main control unit preprocesses the acquired image and spectral data. The visual detection model outputs visual classification labels and confidence scores; the spectral perception model outputs spectral classification labels and confidence scores. The main control unit compares the visual and spectral classification labels; if they are the same, it outputs a common classification result; if they are different, it selects the classification result with the higher confidence score as the final detection result. This invention proposes for the first time a dual-modal confidence score competitive fusion mechanism of "YOLO26 vision + AS7265x spectroscopy," solving the problem of the one-sidedness of single-modal detection and exhibiting strong stability and adaptability in harsh orchard environments.

Description

Technical Field

[0001] This invention belongs to the field of agricultural automation and computer vision technology, and relates to a non-destructive testing system for apple ripeness based on multimodal perception fusion. Background Technology

[0002] Accurate identification of apple maturity is crucial for orchard sorting, cold chain grading, and market pricing. Current mainstream apple maturity identification technologies include visual and spectral detection technologies. While deep learning-based visual detection technologies (such as YOLOv7) have made some progress in apple target recognition, these models are susceptible to interference from uneven lighting, foliage shadows, and overlapping fruits in complex orchard environments with dwarf, dense planting, and high shading. More importantly, visual solutions relying solely on RGB images can only capture the apple's skin color and texture, making it difficult to accurately assess its internal physiological maturity (such as sugar content and starch conversion rate), posing a risk of misjudgment due to "coloring deception." Furthermore, high-performance detection models typically have a large number of parameters and high computational complexity, making it difficult to balance detection accuracy and real-time performance when deployed on resource-constrained edge devices such as Raspberry Pi.

[0003] While laboratory-grade hyperspectral or near-infrared spectrometers can accurately analyze the internal quality of apples, they suffer from drawbacks such as high cost, large size, and complex operation, making them unsuitable for on-site deployment in fields or small grading lines. Portable spectral sensors, while reducing cost and size, are susceptible to interference from external factors such as apple diameter, skin thickness, and ambient temperature, resulting in insufficient robustness of the test results. Although some existing devices have solved the problem of inconvenient adjustment of the conveyor mechanism, their core detection methods remain relatively simple and lack the ability to perform multimodal data fusion analysis and real-time decision-making at the edge.

[0004] In summary, the existing technology lacks a robust non-destructive testing solution for apple ripeness that can operate at low power consumption at the edge, overcome interference from complex environments to achieve high-precision visual positioning, and integrate internal quality spectral information to achieve coordinated judgment of appearance and internal characteristics. Summary of the Invention

[0005] To address the technical challenges of existing technologies where visual inspection and spectral inspection are independent, and high-precision visual models cannot achieve low-latency, high-precision real-time inference on edge devices with limited computing power, a non-destructive testing system for apple maturity based on multimodal perception fusion is provided.

[0006] The technical solution provided by this invention is: a non-destructive testing system for apple ripeness based on multimodal perception fusion. This system includes a camera, a spectral sensor, a main control unit, and a visual detection model and a spectral perception model deployed on the main control unit. The camera and spectral sensor are connected to the main control unit. The main control unit preprocesses the acquired image data and spectral data. The visual detection model extracts visual features from the preprocessed image data and outputs visual classification labels. and confidence level The spectral sensing model extracts spectral features from the preprocessed spectral data and outputs spectral classification labels. and confidence level The main control unit compares the visual classification label with the spectral classification label, if... = This indicates that the two models make the same judgment, and the common classification result is output; if ≠ ,Compare and The numerical value is used to select the classification result with high confidence as the final detection result; the visual detection model is an improved YOLO26 model, which introduces GhostConv in the backbone network; WaveletPool and WaveletUnPool are introduced in the backbone network and neck network; during downsampling, WaveletPool decomposes the feature map into low-frequency approximate components and high-frequency detail components through discrete wavelet transform; during upsampling, WaveletUnPool concatenates and fuses the deep feature map with the high-frequency detail components retained by the shallow network; after each concatenation and fusion operation, a dimensionality reduction operation is performed.

[0007] Preferably, the improved YOLO26 model introduces a C2PSA module after multiple consecutive C3k2 modules at the end of the backbone network; the C2PSA module performs global context modeling of deep features through a spatial attention mechanism.

[0008] Preferably, the improved YOLO26 model introduces the ACmix module into the neck network. After the high-resolution feature layer, medium-resolution feature layer, and low-resolution feature layer complete feature splicing and fusion, the ACmix module is used for mixed feature extraction.

[0009] Preferably, the system further includes a temperature and humidity sensor, which is connected to the main control unit to acquire environmental parameters in real time and provide a compensation factor for the spectral mode sensing type.

[0010] Preferably, the system further includes an interaction module, which adopts a visual operation interface to realize functions such as command issuance, real-time image preview, and detection result display.

[0011] Preferably, the main control unit includes a Raspberry Pi 5, and the Raspberry Pi 5 is connected to an external Hailo-8 AI acceleration module.

[0012] Preferably, the spectral sensing model is an MLP-based spectral sensing model.

[0013] Compared with existing technologies, the present invention has the following significant advantages: (1) High efficiency of parallel heterogeneous computing: This invention combines the general computing power of Raspberry Pi 5 with the accelerated computing power of Hailo-8's 26TOPS neural network. By performing Ghost convolution, C2PSA optimization and strict channel management on the YOLO26 model, it is highly compatible with the pipeline architecture of Hailo-8. Finally, the FPS of an end-to-end inference system for apple maturity detection reaches 30ms, truly realizing low-latency real-time detection at the edge. (2) Breakthrough improvement in visual detection accuracy under complex backgrounds: By introducing the "WaveletPool & WaveletUnPool" wavelet transform pair into the YOLO26 model, this invention overcomes the inherent defect of loss of detail information during downsampling in the traditional YOLO model, enabling the model to capture the most subtle differences in the skin of apples at different maturity levels. Combined with the ACmix module's ability to fuse global and local information, this invention achieves an accuracy of over 95% and an mAP@50 index exceeding 98% in apple maturity detection tasks, significantly outperforming the native YOLO architecture and other existing detection methods; (3) Robustness and accuracy of fusion: This invention proposes and implements for the first time a dual-modal confidence-based competitive fusion framework of "YOLO26 vision + AS7265x spectroscopy" for apple ripeness detection. This framework not only solves the one-sidedness problem of single-modal detection (such as "color deception" or "spectral interference"), but also exhibits strong stability and adaptability in harsh orchard environments such as rainy days and direct sunlight through competitive arbitration logic. Actual test data shows that the accuracy of comprehensive ripeness determination has increased from 89.5% in the traditional monocular vision scheme to 96.2%. (4) Miniaturization and Deployability of the System: The entire system of this invention is integrated into a single Raspberry Pi device and its peripherals, combining visual perception, spectral perception, environmental perception, AI-accelerated computing, and interactive display into a compact housing, resulting in a small overall size and light weight. The system is powered via a Type-C interface, and the built-in battery supports offline operation without the need for complex mechanical installation or site modifications. This design gives this invention significant advantages over large, fixed devices, including low cost, low power consumption, flexible configuration, and immediate usability, making it particularly suitable for direct application in non-industrial settings such as fields, small cooperatives, and fruit wholesale stalls. Attached Figure Description

[0014] Figure 1 This is a hardware architecture modeling diagram of the non-destructive testing system for apple maturity based on multimodal perception fusion according to the present invention. Figure 2 This is a diagram showing the overall network structure of the improved YOLO26 model of this invention. Figure 3 This is a schematic diagram of the ACmix module in the improved YOLO26 model of this invention; Figure 4 This is a schematic diagram illustrating the working principle of WaveletPool in the improved YOLO26 model of this invention; Figure 5 This is a schematic diagram illustrating the working principle of WaveletUnPool in the improved YOLO26 model of this invention; Figure 6 This is a flowchart of the dual-modal confidence competition fusion decision logic in this invention; Figure 7 This is a basic flowchart of the system of the present invention; Figure 8 This is a system structure diagram of the present invention; Figure 9 This paper compares the evaluation results (recall rate) of the improved YOLO26 model and the original YOLO26 model of this invention. Figure 10 This is a comparison of the evaluation results (accuracy) between the improved YOLO26 model and the original YOLO26 model of this invention. Detailed Implementation

[0015] To facilitate understanding of the present invention, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and specific examples. The following examples or drawings are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0016] I. System Overall Architecture and Hardware Integration like Figure 1 As shown, the multimodal perception fusion-based non-destructive testing system for apple ripeness provided in this embodiment integrates its hardware core into a compact housing 5. The housing 5 is integrally molded using 3D printing technology and is made of PLA or ABS engineering plastic, offering good structural strength and portability. An OLED display screen with a resolution of 128×64 pixels is embedded in the front of the housing 5. It communicates with the main control unit via USB to display real-time information such as test results, system status, environmental parameters, and confidence scores. A test button 2 and a power switch 3 are sequentially arranged on the lower panel of the housing 5. The test button 2 is a trigger-type microswitch used to initiate a single test cycle; the power switch 3 is a self-locking power switch used to control the power-on and power-off of the entire system.

[0017] A high-definition USB camera 6 with a physical resolution of 8 megapixels is fixed at the top of the inner cavity of the housing. The lens is mounted vertically downwards, with the optical axis aligned with the center of the apple tray under test. LED supplementary light sources 8 are arranged in a ring around the camera. These light sources use white LED beads with a color temperature of 6500K and are covered with a diffuser plate to ensure uniform and consistent illumination during image acquisition, effectively avoiding strong shadow interference caused by unidirectional light sources. A circular recessed hole is located at the center of the bottom of the inner cavity, housing an AS7265x compact 18-channel spectral sensor 4. This module consists of three cascaded chips: AS72651 (visible light band), AS72652 (near-ultraviolet to visible light band), and AS72653 (near-infrared band), covering a total wavelength range of 410nm to 940nm, with a total of 18 discrete spectral channels. The sensor is mounted with the photosensitive surface facing upwards, at a distance of approximately 5mm to 8mm from the surface of the apple peel. This distance has been experimentally calibrated to obtain the optimal diffuse reflectance spectral signal intensity. A DHT22 temperature and humidity sensor 7 is fixed to the inner wall of the housing. This sensor periodically collects ambient temperature and humidity data at 2-second intervals to provide a real-time temperature compensation factor for the spectral model, correcting the impact of temperature changes on the spectral sensor readings. Multiple grid-like heat dissipation holes 10 are formed on the side of the housing 5 to ensure that the heat generated by the Raspberry Pi 5 and Hailo-8 AI acceleration module during full-load operation can be effectively dissipated, guaranteeing long-term stable operation of the system. A Type-C charging port 9 is located at the bottom of the housing 5 for powering the Raspberry Pi 5 and charging its built-in lithium battery, enabling the device to be used offline and portable.

[0018] The main control unit uses a Raspberry Pi 5 (8GB RAM) as the core controller of the entire system, interacting with various peripherals via a 40-pin GPIO header. The Raspberry Pi 5 connects to a Hailo-8 AI acceleration module via its PCIe 2.0 x1 interface and an M.2 adapter board. This acceleration module has an equivalent half-precision floating-point computing power of 26 TOPS (Tera Operations Per Second), dedicated to forward inference running an improved YOLO26 vision inspection model quantized and compiled using a dedicated toolchain. The specific connection methods for each peripheral are as follows: a high-definition USB camera 6 connects to the Raspberry Pi 5 via a USB 3.0 Type-A interface to achieve high-bandwidth image data transmission; the AS7265x multi-channel spectral sensor 4 and the DHT22 temperature and humidity sensor 7 connect via I... 2 The C bus is connected in parallel to the GPIO2 (SDA) and GPIO3 (SCL) pins of the Raspberry Pi 5, using a 7-bit I / O bus. 2 C address is used for differentiated addressing; OLED display 1 also uses I... 2The system is connected to the C-bus; detection button 2 and switch button 3 are respectively connected to GPIO pins and configured as pull-up inputs, responding to operation commands via interrupts. After the system powers on, the Raspberry Pi 5 automatically loads the pre-installed Python startup script, initializes the drivers for each sensor and the Hailo-8 runtime environment, and then enters standby mode. The operator only needs to place the apple to be tested in the designated detection position inside the casing and press detection button 2. The system will automatically and synchronously trigger the image acquisition and spectral acquisition processes, complete the inference and fusion decision, and then display the maturity detection result on the OLED display screen 1 in real time.

[0019] II. Detailed Implementation Process of the Improved YOLO26 Model Algorithm The visual detection model in this embodiment is an improved YOLO26 model, such as... Figure 2 As shown, the detailed construction steps are as follows: 1. Backbone network optimization: (1) The first layer of the network is a standard convolutional layer (p1). In the second layer of the network (P2), GhostConv (lightweight convolutional layer) is introduced to generate a large number of "shadow" feature maps from a small number of intrinsic feature maps through linear operations. This design significantly reduces the number of parameters and computational cost (FLOPs) in the early stage of the backbone network, making it more suitable for the parallel pipeline architecture of Hailo-8, thereby improving the inference speed without sacrificing the feature representation ability.

[0020] In the initial stage of feature extraction, the first layer (P1) of the network uses standard convolutions to perform preliminary feature extraction on an input RGB image of size 3×640×640, outputting a 64-channel feature map. This is then followed by GhostConv for downsampling and channel expansion to 128. Lightweight convolutional layers generate intrinsic feature maps through a small number of standard convolutions, and then use simple linear transformations to generate Ghost feature maps, thereby reducing computation by 40%-50% without sacrificing accuracy, matching the architecture of Hailo-8.

[0021] (2) Wavelet transform feature preservation strategy: Traditional downsampling operations (such as stride convolution or max pooling) are prone to loss of high-frequency detail information in the image, which is crucial for distinguishing the subtle cyan texture on the surface of “LR_Apple” (low maturity) apples or the slight defects of “HR_Apple” (high maturity).

[0022] This invention introduces WaveletPool (wavelet pooling module) into the backbone and neck network. WaveletPool decomposes the feature map into low-frequency approximation components and high-frequency detail components through Discrete Wavelet Transform (DWT). While halving the spatial size, it completely preserves the high-frequency components that characterize the fine texture of the apple, significantly improving the ability to identify small targets and subtle maturity features.

[0023] like Figure 4 As shown, during the downsampling process, the image is decomposed into low-frequency approximation components (LL) and high-frequency detail components (LH, HL, HH) using Discrete Wavelet Transform (DWT), as shown in the following formula: ; , .

[0024] After the input feature map undergoes a 2-stride convolution operation through each set of filters, it is decomposed into low-frequency approximation components (LL subband), horizontal high-frequency components (LH subband), vertical high-frequency components (HL subband), and diagonal high-frequency components (HH subband). The LL subband retains the macroscopic morphological information of the apple target, such as the overall outline and main color areas; while the three high-frequency subbands (LH, HL, and HH) completely preserve the subtle ripeness texture features of the apple's skin. Traditional pooling or stride convolution directly discards or smooths out these high-frequency details during downsampling, resulting in a decrease in the ability to recognize small targets and subtle features. WaveletPool concatenates the four subbands along the channel dimension (i.e., the number of channels becomes 4 times the original), and then combines this with subsequent 1×1 convolutions for channel compression and information fusion, achieving a halving of spatial resolution while losslessly transmitting high-frequency details to the deep network. Experiments have verified that this strategy improves the model's recall rate for small targets like apples by more than 6%.

[0025] (3) Global Feature Optimization and Numerical Stability: After several C3k2 modules (cross-stage feature fusion modules) at the end of the backbone network, a C2PSA module (fast spatial pyramid pooling module) is introduced. The C2PSA module performs global context modeling of deep features through spatial attention mechanism. While reducing computational complexity, it can effectively prevent the numerical explosion problem that may be caused by the stacking of deep network channels, and ensure the numerical stability and accuracy of the model when performing 8-bit integer quantization on Hailo-8.

[0026] 2. Attention mechanism and feature fusion: (1) Wavelet inverse pooling upsampling: such as Figure 5As shown, during the upsampling process, WaveletUnPool (wavelet unpooling module) is introduced in correspondence with WaveletPool of the backbone network. WaveletUnPool accurately recovers the target boundary and texture information lost due to downsampling by concat (stitching) the deep feature map with the high-frequency detail components retained by the shallow network, which greatly alleviates the problem of missing detection of distant or small target apples.

[0027] (2) An ACmix module (hybrid feature extraction module) is introduced into the neck network. After feature splicing and fusion are completed in the high-resolution feature layer (P3), medium-resolution feature layer (P4), and low-resolution feature layer (P5), the ACmix module is used for hybrid feature extraction. Its internal structure is as follows: Figure 3 As shown, the core innovation of the ACmix module lies in the seamless integration of traditional convolutional operations with a multi-head self-attention mechanism. Its processing flow consists of two stages: Stage I, where the input feature map is projected and mapped using three parallel 1×1 convolutions, generating Query, Key, and Value feature groups corresponding to the standard convolutional path and the self-attention path, respectively; Stage II, where the self-attention path calculates the similarity matrix between the Query and Key, and globally weights and aggregates the Value, capturing long-distance contextual dependencies in the apple image (such as the spatial relationship between the fruit and the branches), effectively suppressing interference from complex textures in the background such as leaves and soil; the convolutional path, through local 3×3 convolutional kernels, finely perceives the color gradations and texture features of the apple skin due to different ripeness. The outputs of the two paths are weighted and fused using learnable weight parameters α and β, ultimately forming an enhanced feature map that combines global contextual awareness and local detail capture capabilities. Experimental data show that after introducing the ACmix module into the neck network, the model's response strength to apple region features is improved, and its detection robustness in complex scenes is significantly enhanced.

[0028] (3) After each Concat concatenation operation, the C3k2 module is used to perform feature dimensionality reduction (e.g., reducing from 1024 channels to 512 channels). This strict feature channel number management strategy effectively controls the memory usage and computational overhead of the model during edge inference, ensuring the stability and efficiency of the inference process.

[0029] (4) Quantization compilation for Hailo-8: The trained PyTorch model is converted to ONNX format and 8-bit log-symmetric quantization is performed using the Hailo dataflow compiler. The activation range of the C2PSA module (fast spatial pyramid pooling layer) is optimized by setting a calibration set to ensure that the accuracy loss of the HEF format model is less than 0.5% when inference on the hardware.

[0030] III. Spectral Feature Extraction and Analysis Based on MLP Data Acquisition and Preprocessing: AS7265x Multichannel Spectrometer 4 via I 2 The system communicates with the Raspberry Pi 5 via the C-bus. At the start of each detection cycle, the system first initializes the sensor using a specific AT command sequence, setting the integration time to 100 milliseconds and the gain to medium. The spectral sensor 4 performs analog-to-digital conversion on the reflected light intensity received by its 18 channels (410nm, 435nm, 460nm, 485nm, 510nm, 535nm, 560nm, 585nm, 610nm, 645nm, 680nm, 705nm, 730nm, 760nm, 810nm, 860nm, 900nm, 940nm) within the integration time, outputting it as 16-bit unsigned integer raw counts.

[0031] Model Structure: The MLP-based spectral sensing model adopts a three-layer MLP architecture (200-100-50). The first input layer receives 18-dimensional spectral features; the hidden layer uses the Tanh activation function and adds a Dropout layer to prevent overfitting; the output layer uses linear output to obtain the raw category score (Logits), and provides accurate judgments for the three levels of LR, MR, and HR through a confidence calibration algorithm.

[0032] IV. System Software Implementation and UI Interaction Flow The software system in this embodiment is based on PyQt5 as the graphical user interface (GUI) framework, and its core logical architecture is as follows: 1. Multithreaded asynchronous execution architecture and task scheduling To address the challenges of edge computing devices in processing high-bandwidth visual streams and low-speed I / O... 2 To address the performance imbalance issue with the C bus, the system employs an asynchronous cooperative architecture. UI main thread logic control: Responsible for managing the lifecycle of the PyQt5 form, receiving data packets pushed by the child thread through the signal and slot mechanism. The main thread embeds a QTimer to refresh the real-time video Label control in the interface every 33ms.

[0033] The Vision Thread is dedicated to managing the HailoRT runtime. This thread employs an asynchronous push model, mapping the weights of the improved YOLO26 model in HEF format to Hailo-8 memory during initialization. In the inference loop, a non-blocking buffer is established using the hailo_rest_api, enabling parallel execution of image preprocessing and model inference, significantly improving the overall system throughput.

[0034] Sensor Thread (Sensor Layer Scheduling): Targeting I 2 Timing is orchestrated for multi-slave addressing on the C bus (AS7265x and DHT22). This thread is in a resident suspended state, initiating atomic-level data read operations only upon receiving a global trigger signal.

[0035] 2. System Workflow State Transitions and Decision Flow: The internal logic of the software is driven by a finite state machine (FSM), focusing on describing the dynamic process from standby to decision-making: Idle Preview (IDLE): The system's main loop continuously captures BGR frames from the camera, converts them to RGB using cv2.cvtColor, and then converts them to QImage for rendering. No complex inference operations are performed during this stage; only real-time monitoring of GPIO interrupt signals is maintained.

[0036] Synchronous sampling trigger logic: Once the system detects an interruption caused by a level change in button 2, it immediately initiates synchronous timestamp alignment: while locking the current visual frame, it sends an integration request to the multispectral sensor.

[0037] Decision-making arbitration execution phase: The decision-making level receives the visual classification labels pushed by the visual thread. and confidence level And the spectral classification tags pushed by the spectral thread. and confidence level Then, the confidence level competition mechanism is automatically executed. If the visual confidence level is attenuated due to environmental occlusion, the arbitration logic will automatically shift the priority to the spectral mode to ensure the consistency of the judgment result.

[0038] 3. PyQt5 User Interface Design The user interface is divided into four core functional areas: Real-time video monitoring area: Located in the center of the interface, it uses OpenCV's cvtColor function to convert BGR frames to RGB and render them, drawing the bounding box of the apple target detected by YOLO26 in real time.

[0039] Spectral Display Area: Located on the right side of the interface, this area displays the 18-channel normalized reflectance values ​​of the AS7265x in real-time numerical form. The differences in reflectance between apples at different stages of ripeness near 680nm (chlorophyll absorption peak) and 900nm are clearly marked.

[0040] Decision information feedback area: Dynamically displays the confidence bars of the visual model and the spectral model respectively.

[0041] Comprehensive judgment results area: Located at the bottom of the interface, it displays the final maturity level (LR_Apple, MR_Apple or HR_Apple) in extra-large font and is accompanied by voice prompts, making it convenient for operators to quickly obtain results in bright light or noisy environments.

[0042] 4. Underlying resource protection and fault tolerance logic I 2 C. Bus Mutual Exclusion Protection Mechanism: Since the AS7265x shares a bus with the OLED display, the software layer uses a mutex lock to achieve exclusive access. During the high-priority cycle of spectral data reading, the system temporarily blocks the OLED's refresh instructions, completely avoiding bus deadlock or communication conflicts.

[0043] V. Bimodal Confidence Competition Fusion Logic like Figure 6 As shown, one of the core innovations of this invention is the design of a dynamic bimodal confidence competition and arbitration decision-making strategy to achieve highly robust judgment by fusing visual and spectral information in complex environments.

[0044] Parallel Inference: After the user clicks the "Start Test" button or presses the detection button 2 on the housing 5 through the PyQt5 visual interface, the system main program creates two parallel running sub-threads using Python's threading or multiprocessing libraries. Thread A (visual thread) uses the OpenCV library to control the high-definition USB camera 6 to capture a frame of RGB image with a resolution of 640×480. After size normalization, it is sent to the Hailo-8 accelerator through the HailoRT API to execute the HEF format improved YOLO26 model inference and return the predicted visual classification label. and its confidence level Thread B (spectral thread) synchronizes via I. 2 The C-bus reads the raw count values ​​from the 18 channels of the spectral sensor 4, feeds them into a lightweight MLP-based spectral sensing model loaded into memory, performs fast forward propagation calculations, and obtains the spectral classification labels. and its confidence level Two threads execute in parallel without blocking each other, minimizing the total inference time. The category label represents apple maturity, and can be LR (eat-ripe, i.e., high maturity), MR (picking-ripe, i.e., medium maturity), or HR (unripe, i.e., low maturity).

[0045] Decision output: If = This indicates that the two models make the same judgment, and outputs this common result; if ≠ ,Compare and The classification label corresponding to the model with higher confidence is selected as the final detection result based on the numerical value of the model.

[0046] This logic ensures that visual ambient light interference ( When the temperature drops, the system will automatically rely on spectral data. The determination is usually made based on stability.

[0047] VI. Overall Basic Process of the Invention This invention revolves around three levels: data acquisition, multi-source information fusion, and visualization interface and application decision-making. Figure 7 As shown.

[0048] First, at the data acquisition layer, a data acquisition platform centered on a Raspberry Pi 5 is built, integrating multiple sensors to achieve real-time acquisition of multi-source environmental data. Specifically, this includes: acquiring apple image data and spectral data through cameras and spectral sensors; and using temperature and humidity sensors to monitor ambient temperature and humidity, providing multi-dimensional environmental auxiliary information for ripeness assessment.

[0049] Secondly, in the multi-source information fusion layer, the system inputs the collected image data and environmental sensor data into the multi-source information fusion model. The heterogeneous data is preprocessed and aligned using a data fusion algorithm, followed by feature extraction. Visual features such as the apple's color, texture, and size are extracted from the images, while environmental features such as temperature, humidity, and light intensity are extracted from the sensor data. Subsequently, a feature fusion algorithm deeply fuses the multimodal features to construct a feature vector that comprehensively reflects the apple's growth status, providing a more accurate basis for maturity determination.

[0050] Finally, at the visualization interface and application decision-making layer, the system develops a user interface that realizes three major functions: data display, data analysis, and decision support. The data display module presents environmental information such as temperature, humidity, weather conditions, and sunshine duration in real time; the data analysis module intelligently identifies and grades apple ripeness; and the decision support module generates predictions of the optimal consumption period and suggestions for purchasing timing based on the analysis results. Through the above technical processes, a complete closed loop is ultimately achieved from multi-source data collection and information fusion processing to visualized decision output, providing scientific advice and results for the refined management of orchards.

[0051] VII. Intelligent Monitoring and Decision-Making Process for Apple Ripeness The overall system architecture of this invention is logically divided from top to bottom into a perception layer, a computing layer, and an interaction layer. Each layer achieves efficient data flow and collaborative control through a standard bus and interface, such as... Figure 8 As shown: Interaction Layer (Human-Computer Collaboration and Result Display): The interaction layer directly faces the end user, providing flexible control input and intuitive information feedback.

[0052] Computation layer (core heterogeneous processing and inference hub): Main controller (Raspberry Pi 5): Responsible for global task scheduling and hardware / software coordination. Its internal logic includes: loading the underlying sensor driver, preprocessing multi-source data, executing spectral MLP forward inference, executing the dual-modal confidence competition fusion algorithm, and maintaining the operation of the backend PyQt5 driver.

[0053] Acceleration unit (Hailo-8 NPU): Receives the image stream after visual preprocessing, runs the visual inference algorithm independently, and sends the category label and confidence score back to the main control unit for final decision arbitration.

[0054] Perception Layer (Underlying Data Acquisition Network): As the core of the system, the perception layer integrates multimodal data acquisition hardware. Various sensors collaborate to acquire raw multimodal data, providing a stable detection lighting environment. This collaborative work ensures the accurate acquisition of raw multimodal data.

[0055] VIII. Experimental Verification and Result Analysis of the System Performance of the Invention Detection Speed: The end-to-end detection process (from pressing the detection button to displaying the result) was timed. The average time for a single complete detection was 32 milliseconds, corresponding to a real-time processing frame rate of approximately 31.2 frames per second (FPS). This includes approximately 4 milliseconds for camera image acquisition and preprocessing, approximately 12 milliseconds for YOLO26 model inference on the Hailo-8, approximately 3 milliseconds for spectral sensor data reading and MLP inference, approximately 1 millisecond for fusion arbitration and interface updates, and the remaining time for system scheduling and I / O waiting. The actual frame rate far exceeds the requirement of real-time detection (30 FPS), fully demonstrating the high efficiency of the architecture proposed in this invention on edge computing devices.

[0056] Accuracy Comparison: As shown in Table 1, experimental data demonstrate that, at the visual level alone, the improved YOLO26 of this invention, by introducing GhostConv and WaveletPool into the backbone network and ACmix and WaveletUnPool into the neck network, improves the maturity classification accuracy by 3.4 percentage points and the mAP@50 index by 2.5 percentage points compared to the native YOLO26. Furthermore, by fusing an MLP-based spectral perception model and a confidence-based competitive decision-making logic, the overall maturity determination accuracy of the system is further improved to 94.7%, representing a further improvement of 3.4 percentage points compared to the previous pure visual approach. These results strongly demonstrate the significant effectiveness and technical superiority of multimodal information fusion and the competitive decision-making framework in the non-destructive detection of apple maturity.

[0057] To verify the effectiveness of the improvement strategy for the YOLO26 visual model in this invention, the accuracy (P / %) and recall (R / %) curves of the model under 300 training batches (Epochs) were extracted for evaluation. Figure 9 and Figure 10 As shown in the comparison curve graph, the red curve represents the original YOLO26 model, and the black curve represents the improved YOLO26 model proposed in this invention, which integrates GhostConv, C2PSA, Wavelet wavelet transform and ACmix, and is represented by YOLO26-GCWA in the figure.

[0058] Accuracy (P) Comparison Analysis: As shown in the accuracy curve, throughout the entire 300-round training cycle, the improved YOLO26 model (black line) significantly outperformed the native YOLO26 model (red line) in both accuracy increase rate and convergence stability. In the early stages of training, the improved YOLO26 model extracted effective features more quickly and broke through high quantiles. In the later stages of training, its accuracy consistently and consistently surpassed that of the native YOLO26 model, reaching a peak of over 0.95. This indicates that the improved YOLO26 model is more accurate in classifying apple maturity features, with a significantly reduced false positive rate.

[0059] Recall (R) Comparison Analysis: The recall curves show that the improved YOLO26 model (black line) demonstrates a significant advantage in overcoming the false negative problem. This is thanks to the WaveletPool / UnPool wavelet transform feature preservation strategy and the introduction of the ACmix module. Reflected in the curves, the black line rapidly climbs to above 0.8 within a very short training batch and maintains a stable high level of around 0.9 in subsequent training. In contrast, the original YOLO26 model (red line) not only has a lower value but also exhibits significant fluctuations. This comparison fully demonstrates the strong robustness and high detection rate of the improved YOLO26 model of this invention.

[0060] Table 1. Comparison of detection results between the improved YOLO26 model and the native YOLO26 model of this invention. .

Claims

1. A non-destructive testing system for apple ripeness based on multimodal perception fusion, characterized in that, The system includes a camera, a spectral sensor, a main control unit, and a visual detection model and a spectral perception model deployed on the main control unit. The camera and spectral sensor are connected to the main control unit. The main control unit preprocesses the acquired image and spectral data. The visual detection model extracts visual features from the preprocessed image data and outputs visual classification labels. and confidence level The spectral sensing model extracts spectral features from the preprocessed spectral data and outputs spectral classification labels. and confidence level The main control unit compares the visual classification label with the spectral classification label, if... = This indicates that the two models make the same judgment, and the common classification result is output; if ≠ ,Compare and The numerical value is used to select the classification result with high confidence as the final detection result; the visual detection model is an improved YOLO26 model, which introduces GhostConv in the backbone network; WaveletPool and WaveletUnPool are introduced in the backbone network and neck network; during downsampling, WaveletPool decomposes the feature map into low-frequency approximate components and high-frequency detail components through discrete wavelet transform; during upsampling, WaveletUnPool concatenates and fuses the deep feature map with the high-frequency detail components retained by the shallow network; after each concatenation and fusion operation, a dimensionality reduction operation is performed.

2. The system according to claim 1, characterized in that, The improved YOLO26 model introduces a C2PSA module after multiple consecutive C3k2 modules at the end of the backbone network; the C2PSA module performs global context modeling of deep features through a spatial attention mechanism.

3. The system according to claim 1, characterized in that, The improved YOLO26 model introduces the ACmix module into the neck network. After the high-resolution feature layer, medium-resolution feature layer, and low-resolution feature layer complete feature splicing and fusion, the ACmix module is used for mixed feature extraction.

4. The system according to claim 1, characterized in that: It also includes a temperature and humidity sensor, which is connected to the main control unit to acquire environmental parameters in real time and provide compensation factors for the spectral sensing model.

5. The system according to claim 1, characterized in that: It also includes an interaction module, which uses a visual operation interface to realize functions such as command issuance, real-time image preview, and detection result display.

6. The system according to claim 1, characterized in that, The main control unit includes a Raspberry Pi 5, which is connected to an external Hailo-8 AI acceleration module.

7. The system according to claim 1, characterized in that, The spectral sensing model is an MLP-based spectral sensing model.

Images