Multi-modal fusion perception and posture regulation and control integrated system for intelligent identification of conopomorpha sinensis

Abstract

The invention provides a multi-modal fusion perception and posture regulation and control integrated system for intelligent identification of conopomorpha sinensis, and relates to the technical field of nondestructive detection of litchi insect pests, and the system comprises a dual-stage posture regulation and control module, a multi-modal fusion perception module and a data fusion and intelligent decision module. According to the method, a two-stage posture correction mechanism integrating passive horizontal posture adjustment and active visual posture adjustment is adopted, high-precision recognition and dynamic adjustment in the pedicel direction can be achieved, the consistency and alignment of samples in multi-modal detection are effectively improved, and then the multi-dimensional feature requirement of pest recognition is met. A multi-source detection mode integrating near infrared spectrum, hyperspectral imaging and X-ray images is adopted, insect pest recognition is carried out from three layers of information of chemical components, surface characterization and internal structures, the complementary feature potential is fully excavated, and finally, a multi-mode collaborative recognition method based on feature selection and a fusion modeling optimization strategy is put forward. And the generalization ability and robustness to different insect pest forms are improved.

Description

Technical Field

[0001] This invention relates to the field of non-destructive testing technology for litchi pests, and in particular to a multimodal fusion sensing and attitude control integrated system for intelligent identification of litchi stem borers. Background Technology

[0002] Lychee, a tropical fruit, is favored by consumers worldwide for its unique flavor and rich nutritional value. In recent years, with the growth of lychee exports and the development of the deep processing industry, the quality, safety, and grading of lychee have gradually attracted widespread attention. Among the many factors affecting lychee quality, the stem borer is the main fruit-boring pest. Its larvae often lurk deep within the fruit stem, making early-stage infestation difficult to identify with the naked eye. This problem seriously affects the quality and market competitiveness of lychee. Traditional pest detection methods mostly rely on manual dissection, which is not only inefficient (less than 200 fruits can be tested per hour) but also has a high fruit loss rate (over 30%), greatly limiting the industrialization of large-scale sorting and export processing. Therefore, developing an efficient, accurate, and automated non-destructive testing technology for lychee pests has become a core technological challenge for the upgrading of the lychee industry.

[0003] Given the highly concealed nature of fruit pests and the complex structure of fruits, spectroscopic and imaging detection technologies offer effective solutions for improving identification accuracy. Near-infrared spectroscopy (NIRS) can reflect changes in internal chemical composition caused by pests; hyperspectral imaging (HSI) provides rich spatial spectral features, helping to identify surface wormholes or color anomalies; while X-ray imaging utilizes density differences to reveal structural defects within the fruit. However, single-modal detection methods have significant limitations: NIRS is easily affected by the curvature of the fruit shape and surface roughness, leading to signal scattering and distortion, causing characteristic fluctuations; hyperspectral imaging has low sensitivity to deep, early-stage pests and a limited signal-to-noise ratio; X-ray imaging, due to the similar density of the fruit pit and pest tissue, is prone to false positives and false negatives. Therefore, single-modal detection methods often fail to meet the requirements of high precision and high stability. To overcome these limitations, multimodal detection has gradually become a research hotspot, and by integrating chemical, appearance, and structural information, it is expected to improve the accuracy and robustness of pest detection.

[0004] However, in multimodal fusion detection, the arrangement of the detection order and the collaborative mechanism between different modalities have a decisive impact on the overall detection effect. To achieve spatial alignment and information fusion of data from different modalities, it is essential to ensure that the sample's posture remains consistent across different sensors. However, in actual automated detection, litchi fruits often experience significant posture changes due to conveying, rolling, and tilting. Random posture changes in the fruit can cause a shift between the sensor's imaging surface and the detection area, leading to signal acquisition fluctuations and feature extraction errors. Existing research has shown that fruit posture has a significant impact on the signal-to-noise ratio and consistency of spectral data.

[0005] While some studies have attempted to initially constrain fruits using mechanical restraint structures or roller mechanisms, these methods are generally suitable for spherical or elliptical fruits. However, for the complex morphology of litchi fruits, which have heart-shaped geometric features (length-to-diameter ratio 1.3–1.5) and warty protrusions (0.8–1.3 mm) on their surface, existing posture control methods still face significant challenges. Therefore, addressing the problems of the complex morphology of litchi fruits, the strong concealment of pests, and the insufficient accuracy and stability of existing single detection modes, this invention proposes a multimodal fusion sensing and posture control integrated system for intelligent identification of litchi stem borers to solve the problems existing in the prior art. Summary of the Invention

[0006] To address the aforementioned problems, the present invention aims to propose a multimodal fusion sensing and posture control integrated system for intelligent identification of litchi fruit borers. This invention not only breaks through the key bottlenecks in existing fruit and vegetable pest detection regarding fruit posture adjustment and multimodal collaborative identification, but also provides a new paradigm for constructing a high-throughput non-destructive testing technology system for complex agricultural products.

[0007] To achieve the objectives of this invention, the invention is implemented through the following technical solution: a multimodal fusion perception and posture control integrated system for intelligent identification of litchi stem borers, including a dual-stage posture control module, used to adjust the spatial posture of litchi fruit during transportation by using passive horizontal posture adjustment and active visual posture adjustment, so that the litchi stem always faces upwards for subsequent detection.

[0008] The multimodal fusion sensing module is used to acquire information data on chemical components, surface characterization and internal structure related to litchi fruit pests through near-infrared spectroscopy, hyperspectral imaging and X-ray imaging methods, respectively.

[0009] The data fusion and intelligent decision-making module is used to integrate multi-source heterogeneous information data obtained through data layer fusion strategy or feature layer fusion strategy, build pest identification model, and make pest identification decisions.

[0010] A further improvement is that the dual-stage attitude control module includes:

[0011] The passive horizontal posture adjustment submodule is used to output lychee fruits in a horizontal posture.

[0012] The active vision posture adjustment submodule is used to identify the direction of the fruit stem through machine vision, adjust the posture using a robotic arm, and output the lychee fruit in an upright posture.

[0013] A further improvement is that the passive horizontal attitude adjustment submodule includes a conveyor belt, and two sets of guide columns symmetrically arranged at a 30° angle are provided above the conveyor belt. Both ends of the guide columns are connected to the frame of the conveyor belt through gaskets, and several sets of gaskets are provided.

[0014] A further improvement is that the active visual pose adjustment submodule includes:

[0015] A visual perception unit is used to acquire the required image data;

[0016] The image processing unit is used to segment images using a pre-trained U-Net model to obtain the positions of the fruit body and stem.

[0017] The robotic arm execution unit is used to execute the control commands of the control unit, and uses a six-axis robotic arm and electric gripper to complete the grasping and upright placement of lychee fruits;

[0018] The control unit is used to perform motion logic and path planning for the robotic arm execution unit based on the acquired positions of the fruit and stem, as well as the overall task scheduling of the active vision posture adjustment submodule.

[0019] A further improvement is that the multimodal fusion sensing module includes:

[0020] The near-infrared spectroscopy submodule is used to detect changes in internal chemical composition caused by pests;

[0021] The hyperspectral imaging submodule is used to acquire spatial spectral information of surface and superficial tissues and identify epidermal damage or color abnormalities.

[0022] The X-ray imaging submodule is used to reveal internal structural defects in litchi based on density differences.

[0023] Further improvements are made in that the data fusion and intelligent decision-making module includes:

[0024] The data preprocessing submodule is used to normalize, reduce noise, and correct the acquired data.

[0025] The data feature extraction submodule is used to extract features from the preprocessed data;

[0026] The data fusion submodule is used to splice and fuse the extracted feature data according to the data layer fusion strategy or the feature layer fusion strategy.

[0027] Further improvements are made in the following ways: The data layer fusion strategy is as follows: After normalizing the data of each modality, they are directly spliced ​​together, and then a unified method is used for preprocessing and feature selection. Finally, a pest identification model is constructed based on the partial least squares regression model.

[0028] Further improvements are made in the following ways: The feature layer fusion strategy is as follows: different methods are used to preprocess and select features for each modality of data, then the resulting feature subsets are concatenated, then dimensionality is compressed by principal component analysis, and finally a pest identification model is constructed based on the partial least squares regression model.

[0029] The further improvement lies in the following: the different methods used for preprocessing and feature selection of each modality's data are as follows:

[0030] Near-infrared spectral data were preprocessed using the standard normal variable transformation method, and then features were extracted using the competitive adaptive reweighted sampling method.

[0031] The hyperspectral imaging data were preprocessed using the Savitzky-Golay smoothing method and the multivariate scattering correction method, and then feature extraction was performed using the competitive adaptive reweighted sampling method.

[0032] Feature extraction was performed on the X-ray imaging data using the grayscale feature method.

[0033] The beneficial effects of this invention are as follows:

[0034] (1) This invention corrects the posture of litchi by using passive horizontal posture adjustment and active visual posture adjustment, and realizes high-precision identification and dynamic adjustment of the direction of litchi stem. It can effectively solve the problem of inconsistent posture caused by random posture changes in fruit during assembly line detection, thereby ensuring the posture stability of litchi in front of multiple detection sensors and greatly improving the consistency and alignment of samples in different modal detection.

[0035] (2) Based on the multi-dimensional feature requirements of pest identification, this invention constructs a multi-modal fusion sensing module that integrates near-infrared spectroscopy, hyperspectral imaging and X-ray images. By performing multi-level detection on litchi fruits, including chemical composition analysis, surface feature recognition and internal structure revelation, it can comprehensively obtain pest-related information, thereby significantly improving the accuracy and stability of pest identification.

[0036] (3) This invention integrates multi-source heterogeneous information data obtained through data layer fusion strategy or feature layer fusion strategy. By introducing feature alignment mechanism and model fusion algorithm, it effectively integrates feature information under different modalities, optimizes the fusion effect of data of each modality, improves the generalization ability and robustness of the detection system for different pest morphologies, and avoids the false detection and false detection problems in single modality detection methods.

[0037] (4) The non-destructive testing technology used in this invention has significant efficiency and accuracy, greatly improving the processing speed and accuracy of pest detection, and breaking through the bottleneck of low efficiency and high fruit damage rate of traditional manual dissection. By reducing fruit damage, this system not only improves the overall efficiency of litchi sorting, but also provides a new paradigm for building a high-throughput non-destructive testing technology system for complex agricultural products. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the system structure of the present invention.

[0039] Figure 2 This is a schematic diagram comparing data layer fusion and feature-level fusion of the present invention.

[0040] Figure 3 This is a schematic diagram of the segmentation model process of the present invention.

[0041] Figure 4 This is a schematic diagram of the workflow of the active visual posture adjustment submodule of the present invention.

[0042] Figure 5 This is a top view schematic diagram of the passive horizontal attitude adjustment submodule structure of the present invention.

[0043] The components include: 1. Conveyor belt; 2. Guide column; 3. Gasket. Detailed Implementation

[0044] To enhance understanding of the present invention, the present invention will be further described in detail below with reference to embodiments. These embodiments are only used to explain the present invention and do not constitute a limitation on the scope of protection of the present invention.

[0045] according to Figures 1-5 As shown, this embodiment proposes a multimodal fusion perception and posture control integrated system for intelligent identification of litchi stem borers. It includes a two-stage posture control module, used during transport to adjust the spatial posture of the litchi fruit through passive horizontal posture adjustment and active visual posture adjustment, ensuring the litchi stem always faces upwards for subsequent detection. The passive horizontal posture adjustment fully utilizes the differences in density and geometric characteristics between the litchi stem and the fruit body. During conveyor belt movement, it automatically controls the litchi's posture through structural guidance and centrifugal force coupling, ensuring the fruit reaches a stable state that meets identification requirements before entering the detection area.

[0046] Specifically, the two-stage attitude control module includes:

[0047] A passive horizontal posture adjustment submodule, used to output lychee fruits in a horizontal posture, includes a conveyor belt 1. Above the conveyor belt 1 are two sets of guide posts 2 symmetrically arranged at a 30° angle. Both ends of the guide posts 2 are connected to the frame of the conveyor belt 1 via shims 3. Several sets of shims 3 are provided. The two sets of guide posts 2 form a V-shaped guiding mechanism, thus creating a rolling channel. During this process, the vertical height difference of 1–4 mm can be adjusted by changing the number of shims 3 (adjusted by stacking the shims 3 vertically, and their connection is secured with bolts) to accommodate different lychee sizes. Furthermore, conveyor belt 1 uses a 100mm wide PU material belt, driven by a servo motor, with a belt speed adjustable within the range of 0.1–1.0 m / s. During operation, lychees are fed by an external vibrating feeder, causing them to fall evenly onto conveyor belt 1 and enter the correction channel of the V-shaped guide mechanism in a random manner. Under the physical constraint of the V-shaped structure, the lychees roll along the channel. Simultaneously, due to the high density in the stem area (approximately 1.05–1.15 g / cm³), the lychees... 3 Under the combined action of centrifugal force and geometric properties, its axial direction tends to automatically align with the direction of the conveyor belt, thereby achieving initial posture correction.

[0048] An active visual posture adjustment submodule is used to identify the direction of the fruit stem through machine vision, adjust the posture using a robotic arm, and output the lychee fruit in an upright posture. It includes:

[0049] The visual perception unit is used to acquire the required image data, which is completed by the corresponding RGB-D camera. The image resolution is 640×480, and the acquisition distance is controlled between 20 and 30cm.

[0050] The image processing unit is used to segment images using a pre-trained U-Net model to obtain the positions of the fruit body and stem. The U-Net model structure adopts a typical encoder-decoder symmetric framework. The encoder part introduces a ResNet backbone network and removes the fully connected classification layer to enhance the image feature extraction capability. It also combines a multi-scale feature fusion mechanism and a residual nesting structure to improve robustness to complex backgrounds and pose changes. The decoder fuses shallow high-resolution features and deep semantic information from the encoder through skip connections to achieve a synergistic enhancement of boundary preservation and segmentation accuracy.

[0051] The robotic arm execution unit is used to execute the control commands of the control unit, and uses a six-axis robotic arm and electric gripper to complete the grasping and upright placement of lychee fruits;

[0052] The control unit is used to perform motion logic and path planning for the robotic arm execution unit based on the acquired positions of the fruit and stem, as well as the overall task scheduling of the active vision posture adjustment submodule.

[0053] In the motion process, the trained U-Net model is first used for image segmentation to obtain the positions of the fruit and its stem. Then, the centroid of the fruit is extracted as the grasping target, and combined with the stem's direction vector, pose determination is completed. The robotic arm uses MoveIt 2 (based on the ROS2 framework) for motion path planning and control execution, completing the precise grasping and upright placement of the lychee. This ensures that the lychee stem always faces upwards, providing a foundation for spatial consistency acquisition in multimodal detection. Figure 4 As shown, after system initialization, the robotic arm first returns to the preset initial pose and completes spatial calibration with the camera's viewpoint. Then, the top RGB-D camera acquires a planar image of the worktable and inputs it into the U-Net network to complete fruit semantic segmentation, obtaining masks for the stem and fruit body regions. Image processing calculates the centroid of each region and constructs direction vectors to determine the stem orientation. Then, using the camera's intra-camera depth map information, it converts this into a grasping point and correction direction in a 3D coordinate system. Based on this, the system performs trajectory planning, controlling the robotic arm to move to the pre-grasping pose and slowly closing the gripper to complete the grasping operation. To meet standardized correction requirements, the system introduces end-effector posture torsion logic, allowing the lychee to adjust its posture after grasping, ensuring the stem faces upwards. Finally, the robotic arm transfers the lychee to the designated tray area, releases the fruit, and returns to the initial waiting position. The system then enters the next round of recognition and processing, establishing a stable automated operation cycle.

[0054] The multimodal fusion sensing module is used to acquire information data on chemical components, surface characterization, and internal structure related to litchi fruit pests through near-infrared spectroscopy, hyperspectral imaging, and X-ray imaging methods, including:

[0055] The near-infrared spectroscopy submodule is used to detect changes in internal chemical composition caused by pests;

[0056] The hyperspectral imaging submodule is used to acquire spatial spectral information of surface and superficial tissues and identify epidermal damage or color abnormalities.

[0057] The X-ray imaging submodule is used to reveal internal structural defects in litchi based on density differences.

[0058] The data fusion and intelligent decision-making module is used to integrate multi-source heterogeneous information data acquired through data-layer fusion strategies or feature-layer fusion strategies, construct pest identification models, and make pest identification decisions. It includes:

[0059] The data preprocessing submodule is used to normalize, reduce noise, and correct the acquired data.

[0060] The data feature extraction submodule is used to extract features from the preprocessed data;

[0061] The data fusion submodule is used to splice and fuse the extracted feature data according to the data layer fusion strategy or the feature layer fusion strategy.

[0062] The data fusion strategy is as follows: after normalizing the data of each modality, they are directly spliced ​​together, and then a unified method is used for preprocessing and feature selection. Finally, a pest identification model is constructed based on the partial least squares regression model.

[0063] In the data fusion strategy, the original spectral and image data are first normalized. Then, the normalized data are concatenated column-wise to form the fused dataset. This step ensures that different data sources are compared on a uniform scale, reducing bias caused by different units of measurement. Next, standardization (SNV) is applied to the fused data to eliminate differences in units of measurement between different features, making model training more stable. In the feature extraction stage, the Competitive Adaptive Reweighted Sampling (CARS) method is used to select the most representative and effective features from the fused data. Finally, partial least squares regression (PLSR) is used for modeling, and accuracy is used as the evaluation metric to assess the model's performance, as shown in Table 1 below.

[0064] Table 1. Performance Comparison of Different Feature Methods under Optimized Preprocessing

[0065]

[0066] Experimental results show that data layer fusion, combined with SNV preprocessing and CARS feature extraction methods and PLSR modeling, achieves optimal performance. The test set accuracy reached 95.82%, and the training set accuracy reached 96.67%, demonstrating the effectiveness and superiority of this strategy in processing multi-source data.

[0067] The feature layer fusion strategy is as follows: different methods are used to preprocess and select features for each modality of data, then the resulting feature subsets are concatenated, and then dimensionality is compressed by principal component analysis. Finally, a pest identification model is constructed based on a partial least squares regression model.

[0068] The following methods are used to preprocess and select features for each modality of data:

[0069] Near-infrared spectral data were preprocessed using the standard normal variable transformation method, and then features were extracted using the competitive adaptive reweighted sampling method.

[0070] The hyperspectral imaging data were preprocessed using the Savitzky-Golay smoothing method and the multivariate scattering correction method, and then feature extraction was performed using the competitive adaptive reweighted sampling method.

[0071] Feature extraction was performed on the X-ray imaging data using the grayscale feature method.

[0072] In the feature-level fusion strategy, features are first extracted from each data source, and then selected and combined to form a comprehensive feature vector. For visible and near-infrared spectral data, SNV and CARS methods are used to extract features; for hyperspectral data, SG, MSC, and CARS are used; and for image data, grayscale feature extraction methods are used. After feature extraction, features from different data sources are selected and combined to form a comprehensive feature vector. When selecting features, the correlation and redundancy of features are considered to avoid excessive redundant information negatively impacting model performance. After feature-level fusion, the comprehensive feature vector is preprocessed again. By calculating the correlation coefficient between features, highly correlated features are removed, and highly independent features are retained to further improve model performance. The PLSR model is used to model the optimized feature vector, and accuracy is used as the evaluation metric. (See Table 2 below.)

[0073] Table 2 Comparison of Results from Different Feature Selection Methods

[0074]

[0075] The results show that feature-level fusion, after fusing the features extracted by each detection method under the corresponding optimal data preprocessing and feature extraction methods, and then performing preprocessing and modeling, also performs excellently, achieving an accuracy of 95% on the test set and 95.40% on the training set. In-depth analysis of the results of the two fusion strategies reveals that both feature-level fusion and data-level fusion demonstrate significant efficiency in processing multi-source information, achieving an accuracy of 96.67% on the test set, showing comparable performance. Feature-level fusion, by accurately selecting and integrating key features from various data sources, maximizes the complementarity of different modalities, especially when processing data with different dimensions and properties, enabling more effective information integration and improving model accuracy and stability. Data-level fusion, on the other hand, achieves preliminary information integration during the data preprocessing stage, suitable for scenarios with high similarity and consistency between data sources, and can more efficiently preserve the complete structure of the original data, providing a solid foundation for subsequent modeling. Therefore, both data-level fusion and feature-level fusion provide effective strategies for multi-source information fusion. In practical applications, these two fusion methods can be flexibly selected or combined according to the specific application scenario, data characteristics, and specific requirements for model performance in order to achieve the best detection results.

[0076] For the passive horizontal attitude adjustment submodule, lychees exhibit significant irregularities in their natural state, with abrupt changes in morphology and differences in contact angle between the stem and shoulder, resulting in a noticeable shift in center of gravity during rolling. When lychees enter the system via a vibrating feeder, their initial posture is typically randomly distributed. After entering a V-shaped guide channel composed of two stainless steel tubes, this invention uses angled guidance coupled with the gravity component to cause the lychees to undergo restricted rolling along the channel axis under the drive of the conveyor belt. At this point, the lychees in the V-shaped symmetrical structure are supported not only by the normal force of the tube wall but also by the component force in the tangential direction of the tube, thereby inducing active rolling behavior.

[0077] Under the overall inertia caused by the uniform motion of the conveyor belt, the center of mass of the lychee interacts with the geometric constraints of the V-shaped structure, ensuring its rolling direction remains stable along the conveyor direction. Simultaneously, the stem end, due to its higher density, becomes heavier and tends to move closer to the bottom of the V-groove under the influence of centrifugal effect and rolling friction, gradually adjusting its axis to be parallel to the conveyor belt's running direction, thus completing the dominant posture adjustment. Throughout this process, automatic correction is achieved based on a completely passive structure, requiring no additional sensor intervention or active control strategies. This ensures structural simplicity while achieving a high posture standardization rate, making it particularly suitable for online detection environments with large-volume continuous feeding. Furthermore, the system's overall structural parameters (such as gap length, guide height, and belt speed) have excellent adjustability, enabling rapid adaptation and dynamic optimization based on the physical properties of the target fruit, demonstrating good versatility and scalability.

[0078] For the active visual pose adjustment submodule, a fruit body and stem segmentation model based on an improved U-Net is used, and its recognition accuracy and inference stability are verified in real-world scenarios. The model adopts an encoder-decoder symmetrical structure, with a ResNet residual network introduced into the backbone to enhance the expression of deep features, and the classification layer is removed to adapt to pixel-level prediction tasks. Through nested multi-scale feature fusion and skip connection mechanisms, the model takes into account both global semantics and boundary detail representation, significantly improving the segmentation accuracy of the litchi fruit body and stem region.

[0079] During the training phase, the model was trained and validated using 813 image samples annotated in PascalVOC format. Image enhancement techniques included rotation, noise addition, and illumination changes to improve model robustness. On the test set, the model achieved a mean intersection-over-union (mIoU) of 92.67%, and mean precision and recall both reached 96%, demonstrating excellent segmentation performance. The model's loss value steadily decreased during training and eventually converged, validating the effectiveness of the improved structure in avoiding gradient vanishing and improving convergence speed. The specific segmentation model process and performance are as follows: Figure 3 As shown, where Figure 3 In the diagram, 'a' represents a schematic diagram of the net model process, 'b' represents a schematic diagram of model performance, and 'c' represents a schematic diagram of the loss curve.

[0080] In the application phase, an RGB-D camera acquires lychee images in real time at a acquisition distance of 20–30 cm, and the model completes the segmentation and mask generation of the fruit body and stem region. Based on the segmentation results, the system further extracts the centroid coordinates of the two and calculates the direction vector of their connecting line, thereby estimating the stem orientation. Experimental results show that the visual recognition system can complete the pose recognition and localization of a single lychee within an average of 1 second, a significant improvement over the initial version, and is suitable for real-time calibration requirements in production lines.

[0081] Integrating near-infrared spectroscopy (NIRS), hyperspectral imaging (HSI), and X-ray three-dimensional detection technologies, and designing a two-stage attitude correction mechanism that combines passive guidance of a V-shaped circular tube with visual servoing of a ROS robotic arm, the attitude standardization and data spatial alignment of complex fruits in the multimodal detection process were realized.

[0082] Experimental results show that the horizontal and vertical posture correction methods designed in this invention achieve correction success rates of 94.23% and 91.11%, respectively, effectively mitigating the impact of posture variations on signal quality and improving detection accuracy and stability. In single-modal detection, NIRS combined with the SNV+CARS-PLSR model achieves a recognition accuracy of 90.00% in vertical posture, HSI after SG+CARS-PLSR processing reaches 93.33% in lying posture, and X-ray detection combined with grayscale features and an SVM classifier achieves a recognition performance of 85.00%. Furthermore, a multi-modal fusion model constructed based on a data layer fusion strategy further improves the detection accuracy to 96.67%, an improvement of 3.34 percentage points compared to the optimal single-modal method, significantly enhancing the system's recognition robustness and stability.

[0083] The integrated framework of "posture standardization - multimodal perception - fusion modeling" constructed in this invention provides a novel technical path and theoretical support for efficient, intelligent and non-destructive pest detection of lychee and other complex fruits, and has high engineering application value and promotion potential.

[0084] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the present invention without departing from its framework and scope of application, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal fusion sensing and attitude control integrated system for intelligent identification of litchi stem borers, characterized in that: It includes a dual-stage posture control module, which uses passive horizontal posture adjustment and active visual posture adjustment during the transportation process to adjust the spatial posture of the lychee fruit so that the lychee stem always faces upwards for subsequent detection. The multimodal fusion sensing module is used to acquire information data on chemical components, surface characterization and internal structure related to litchi fruit pests through near-infrared spectroscopy, hyperspectral imaging and X-ray imaging methods, respectively. The data fusion and intelligent decision-making module is used to integrate multi-source heterogeneous information data obtained through data layer fusion strategy or feature layer fusion strategy, construct pest identification model, and make pest identification decisions.

2. The multimodal fusion sensing and attitude control integrated system for intelligent identification of litchi stem borers according to claim 1, characterized in that: The two-stage attitude control module includes: The passive horizontal posture adjustment submodule is used to output lychee fruits in a horizontal posture. The active vision posture adjustment submodule is used to identify the direction of the fruit stem through machine vision, adjust the posture using a robotic arm, and output the lychee fruit in an upright posture.

3. The multimodal fusion sensing and attitude control integrated system for intelligent identification of litchi stem borers according to claim 1, characterized in that: The passive horizontal posture adjustment submodule includes a conveyor belt machine (1). Two sets of guide columns (2) are symmetrically arranged at a 30° angle above the conveyor belt machine (1). Both ends of the guide columns (2) are connected to the frame of the conveyor belt machine (1) through gaskets (3). Several sets of gaskets (3) are provided.

4. The multimodal fusion sensing and attitude control integrated system for intelligent identification of litchi stem borers according to claim 1, characterized in that: The active visual posture adjustment submodule includes: A visual perception unit is used to acquire the required image data; The image processing unit is used to segment images using a pre-trained U-Net model to obtain the positions of the fruit body and stem. The robotic arm execution unit is used to execute the control commands of the control unit, and uses a six-axis robotic arm and electric gripper to complete the grasping and upright placement of lychee fruits; The control unit is used to perform motion logic and path planning for the robotic arm execution unit based on the acquired positions of the fruit and stem, as well as the overall task scheduling of the active vision posture adjustment submodule.

5. The multimodal fusion sensing and attitude control integrated system for intelligent identification of litchi stem borers according to claim 1, characterized in that: The multimodal fusion sensing module includes: The near-infrared spectroscopy submodule is used to detect changes in internal chemical composition caused by pests; The hyperspectral imaging submodule is used to acquire spatial spectral information of surface and superficial tissues and identify epidermal damage or color abnormalities. The X-ray imaging submodule is used to reveal internal structural defects in litchi based on density differences.

6. The multimodal fusion sensing and attitude control integrated system for intelligent identification of litchi stem borers according to claim 1, characterized in that: The data fusion and intelligent decision-making module includes: The data preprocessing submodule is used to normalize, reduce noise, and correct the acquired data. The data feature extraction submodule is used to extract features from the preprocessed data; The data fusion submodule is used to splice and fuse the extracted feature data according to the data layer fusion strategy or the feature layer fusion strategy.

7. The multimodal fusion sensing and attitude control integrated system for intelligent identification of litchi stem borers according to claim 1, characterized in that: The data layer fusion strategy is as follows: after normalizing the data of each modality, they are directly spliced ​​together, and then a unified method is used for preprocessing and feature selection. Finally, a pest identification model is constructed based on the partial least squares regression model.

8. The multimodal fusion sensing and attitude control integrated system for intelligent identification of litchi stem borers according to claim 1, characterized in that: The feature layer fusion strategy is as follows: different methods are used to preprocess and select features for each modality of data, then the resulting feature subsets are concatenated, then dimensionality is compressed by principal component analysis, and finally a pest identification model is constructed based on partial least squares regression model.

9. The multimodal fusion sensing and attitude control integrated system for intelligent identification of litchi stem borers according to claim 8, characterized in that: The preprocessing and feature selection methods for each modality of data are as follows: Near-infrared spectral data were preprocessed using the standard normal variable transformation method, and then features were extracted using the competitive adaptive reweighted sampling method. The hyperspectral imaging data were preprocessed using the Savitzky-Golay smoothing method and the multivariate scattering correction method, and then feature extraction was performed using the competitive adaptive reweighted sampling method. Feature extraction was performed on the X-ray imaging data using the grayscale feature method.

Images