Agricultural product quality identification method and system based on multi-modal fusion and feature enhancement

Abstract

The invention belongs to the technical field of agricultural product automatic detection, and discloses an agricultural product quality identification method and system, and the method comprises the steps: synchronously obtaining a visible light image, a structured light reflection image, a hyperspectral image and other multi-mode images; constructing a three-dimensional surface model by using the structured light phase information, and mapping the multi-source data onto the three-dimensional surface model to form a unified data volume; self-adaptive fusion is carried out through a cross-modal attention network, and depth features are generated; secondly, augmenting features of rare defects in a feature space by adopting a conditional generative adversarial network to solve the problem of data scarcity; finally, a classifier is trained based on the augmented data set, high-precision recognition of the quality of the agricultural products is achieved, and the robustness and the recognition breadth of complex scenes are remarkably improved. According to the method, multi-source information is deeply integrated, the problem of data scarcity is innovatively solved, and the breadth and precision of agricultural product quality identification are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of automated agricultural product detection, and in particular to a method and system for non-invasive agricultural product quality identification using multimodal imaging technology and deep learning algorithms. Background Art

[0002] With consumers' increasing demands for agricultural product quality and the continued rise in labor costs, automated, non-destructive, and high-precision agricultural product quality detection technology has become a key link in modern agriculture and food processing industries. Detection methods based on machine vision and artificial intelligence have attracted widespread attention due to their efficiency and objectivity. However, existing technologies still face many challenges in practical applications, resulting in their recognition accuracy and robustness being unable to meet the needs of large-scale commercial applications.

[0003] The limitations of existing technologies are mainly reflected in the following aspects: First, technologies that rely on a single imaging modality have limitations in their ability to acquire information. Traditional visible light, or RGB, imaging technology, while able to effectively capture external surface features of agricultural products such as color, shape, size, and surface texture, is powerless for many key internal or sub-surface quality indicators. For example, fruits such as apples and pears are very prone to early bruises during post-harvest processing. This type of damage occurs beneath the peel and initially shows no color change on the surface, making it impossible for RGB cameras to detect it.

[0004] Similarly, some early decay caused by fungi or changes in internal chemical composition, such as changes in the sugar-acid ratio and soluble solids content, cannot be assessed using RGB images; to overcome this limitation, researchers have developed structured light reflectance imaging (SIRI) technology and hyperspectral imaging (HSI) technology.

[0005] SIRI technology is sensitive to changes in the physical structure of tissues and can penetrate the peel to detect early bruises, but its ability to detect quality problems caused by changes in chemical composition is limited. HSI technology excels at chemical composition analysis, but usually has problems such as slow acquisition speed, large data processing volume, high cost, and susceptibility to the curvature and geometry of the object surface.

[0006] Secondly, existing multimodal data fusion strategies are usually rudimentary and inefficient. In order to combine the advantages of different sensors, researchers have begun to explore multimodal data fusion solutions.

[0007] However, most current methods only use a simple feature layer splicing method to directly connect the feature vectors extracted from different modalities. This method does not consider the complex nonlinear correlation between the modal information, nor can it adaptively adjust the weights of different modalities according to the specific detection task or defect type. It may not only introduce redundant and noisy information, but may even pollute the entire feature space due to low-quality data of a certain modality, ultimately leading to a decline in classification performance.

[0008] Thirdly, the processing and utilization of three-dimensional geometric information is still insufficient. When performing two-dimensional imaging on spherical or quasi-spherical agricultural products, distinguishing true surface defects from normal, naturally occurring depressions, such as the fruit stalk or calyx pit, has been a long-standing challenge. These depressions often appear as dark spots on two-dimensional images due to lighting and shadow effects, making them easily misidentified as defects and leading to a high false positive rate.

[0009] Although some studies have attempted to address this problem by reconstructing the three-dimensional surface of objects through multi-view imaging or structured light technology, these methods are usually used as an independent preprocessing step and fail to be deeply integrated with multimodal data analysis to form a unified, geometrically corrected data representation framework.

[0010] Finally, the data scarcity problem of rare defect samples is widespread. The performance of deep learning models depends heavily on large-scale and diverse training datasets.

[0011] In the scenario of agricultural product quality inspection, normal or common grade samples are easy to obtain, but certain specific defects with significant economic value, such as early rot, specific lesions, internal hollowness, etc., occur very rarely under natural conditions, making it difficult to collect a sufficient number of samples to effectively train deep learning models. This problem seriously restricts the detection system's ability to identify these rare but critical defects.

[0012] In summary, existing technologies are either limited by their reliance on a single information source, or inefficient due to poor data fusion and representation. They are also generally plagued by a shortage of rare defective samples. Therefore, a new approach is urgently needed that can deeply fuse multi-source sensory information, effectively process three-dimensional geometry, and intelligently address data scarcity issues, in order to achieve comprehensive, accurate, and robust identification of agricultural product quality. Summary of the Invention

[0013] The purpose of the present invention is to overcome the defects of the prior art such as incomplete information acquisition, low data fusion efficiency and insufficient rare defect samples caused by reliance on a single information source, resulting in low recognition accuracy and poor robustness, and to provide an agricultural product quality identification method and system based on multimodal fusion and feature enhancement that can comprehensively, accurately, robustly and efficiently complete non-destructive testing of agricultural product quality.

[0014] To achieve the above object, the present invention provides the following technical solutions: A method and system for agricultural product quality identification based on multimodal fusion and feature enhancement is proposed. This method achieves a comprehensive assessment of agricultural product quality by collaboratively collecting multi-source information, constructing a unified three-dimensional representation, using an intelligent fusion network, and performing data augmentation in the feature space. The technical solution includes the following core steps: First, a collaborative multimodal sensing system is used to synchronously collect visible light information, structured light reflectance information, and hyperspectral information of the agricultural products to be tested. Secondly, a unified three-dimensional data representation model for agricultural products is constructed. Thirdly, a cross-modal attention gated fusion network is used to deeply fuse multi-source information to generate a feature vector with strong discriminative power. Finally, for rare defect categories, data augmentation is performed in the feature space to optimize the classifier performance.

[0015] Preferably, the method for constructing a unified three-dimensional data representation model of agricultural products is to innovatively utilize phase data in structured light reflection information to accurately reconstruct the three-dimensional surface geometric model of the agricultural products.

[0016] This three-dimensional model is not only used to identify and eliminate confusion caused by natural geometric structures such as the stalk or calyx, but is also used as a common coordinate system to accurately align and map all other sensor information, including visible light images, structured light reflection intensity information and hyperspectral data, thereby generating a unified multimodal data body with complete information and geometric correction.

[0017] Further preferably, the method of performing deep fusion using a cross-modal attention gated fusion network is to design a novel deep learning network structure that can imitate the decision-making process of domain experts.

[0018] It dynamically learns the intrinsic correlation between features of different modalities through a cross-modal attention mechanism, and uses a gating unit to adaptively perform weighted fusion of multimodal information based on the learned attention weights. This design enables the network to intelligently focus on the most relevant sensor information for specific detection tasks or defect types, thereby generating deep fusion features with high information density and strong discriminative power.

[0019] More preferably, the method for performing data augmentation in the feature space is to use a conditional generative adversarial network (cGAN) to solve the problem of insufficient rare defect samples.

[0020] This approach bypasses the significant challenge of directly generating physically realistic multimodal images, instead operating directly within the deeply fused feature space generated by the aforementioned fusion network. Through training, the cGAN learns and grasps the characteristic distribution of rare defect categories, thereby generating a large number of high-quality synthetic deeply fused feature vectors that are statistically consistent with real samples. This is used to expand the training set, significantly enhancing the final classifier's ability to identify rare defects.

[0021] Compared with the prior art, the present invention has the following beneficial effects: The method of the present invention can not only detect multiple defect types that cannot be covered by a single technology, ranging from early sub-surface physical scratches to internal chemical composition anomalies, but also significantly improves the accuracy and robustness of recognition through intelligent fusion and three-dimensional correction; in particular, its innovative feature space data augmentation technology provides an effective and feasible new paradigm for solving the data imbalance problem that is prevalent in industrial visual inspection, enabling the model to have an unprecedented high detection rate for rare but critical defect categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for describing the embodiments.

[0023] Figure 1 Schematic diagram of the overall architecture of the agricultural product quality identification system of the agricultural product quality identification method and system based on multimodal fusion and feature enhancement of the present invention; Figure 2 A detailed flow chart of an agricultural product quality identification method and system based on multimodal fusion and feature enhancement of the present invention; Figure 3 This is a schematic diagram of the structure of the cross-modal attention gated fusion network CMAG-Net in the agricultural product quality identification method based on multimodal fusion and feature enhancement of the present invention; Figure 4 This is a conceptual diagram of the feature space data augmentation process based on the conditional generative adversarial network (cGAN) in the agricultural product quality identification method based on multimodal fusion and feature enhancement and the system of the present invention; Figure 5 This is a visualization example diagram of unified three-dimensional multimodal data mapping and defect detection results for an agricultural product to be tested, specifically an apple, in the agricultural product quality identification method and system based on multimodal fusion and feature enhancement of the present invention. DETAILED DESCRIPTION

[0024] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0025] Example 1 See also Figure 1 As shown, this embodiment provides an agricultural product quality identification system, which mainly includes a conveying device 101, a multimodal imaging module 102 and a central processing unit 103.

[0026] The conveying device 101 can be specifically a conveyor belt, which is used to transport the agricultural products to be tested at a controllable speed. These agricultural products can be apples, pears, or citrus fruits. A position encoder is installed on the side of the conveyor belt. The encoder is used to accurately track the position of the agricultural products and provide a synchronous trigger signal for the multimodal imaging module 102.

[0027] The multimodal imaging module 102 is installed above the conveyor 101 and is designed to simultaneously acquire multimodal images of each agricultural product. The module specifically includes a visible light RGB camera 102a, which is a standard industrial area array camera used to capture the color and texture information of the surface of the agricultural product.

[0028] The structured light reflection imaging SIRI system consists of a digital micromirror device (DMD) projector 102b and a high-speed monochrome camera 102c. The DMD projector is responsible for projecting a sinusoidal stripe pattern of a preset spatial frequency onto the surface of agricultural products, while the monochrome camera is used to capture the stripe image after reflection from the surface of agricultural products.

[0029] The hyperspectral HSI camera 102d is a linear array hyperspectral camera that uses the push-broom imaging principle and is used to obtain continuous spectral information of each point on the surface of agricultural products as they pass through the camera's field of view on a conveyor belt. The operating wavelength range is, for example, 400 to 1000 nanometers.

[0030] The data collection process corresponds to Figure 2The process for step S201 in Figure 1 is as follows: When an agricultural product enters the imaging area, the position encoder sends a trigger signal. Upon receiving the signal, the RGB camera 102a, the SIRI system, and the HSI camera 102d are synchronously activated. The RGB camera captures a complete two-dimensional color image. The SIRI system rapidly projects at least three sinusoidal patterns with a phase difference of two-thirds of a radian, and the monochrome camera synchronously captures three corresponding reflected images. At the same time, the HSI camera continuously collects line spectrum data. As the agricultural product moves, the data is eventually spliced ​​into a complete hyperspectral data cube.

[0031] Through precise hardware synchronization and spatial calibration, it is possible to ensure that the data of these three different modalities have precise spatial correspondence at the pixel level, and all collected raw data are transmitted to the central processing unit 103 in real time for subsequent processing.

[0032] Example 2 This embodiment describes in detail how to convert the collected multi-source heterogeneous data into a unified, information-complete analysis object, corresponding to Figure 2 Step S202 in .

[0033] Step 201, 3D surface model reconstruction: The central processing unit 103 first processes the three frames of phase-shifted fringe images collected by the SIRI system; based on the principle of spatial frequency domain imaging, the DC component DC image, AC component AC image and phase image of each pixel point can be demodulated from these three frames of images.

[0034] Among them, the DC image is similar to an ordinary grayscale image taken under uniform illumination, reflecting the average reflectivity of the surface; the AC image carries depth-resolved information related to the scattering of light inside the tissue and is particularly sensitive to sub-surface defects; and the phase image directly encodes the three-dimensional height information of the object surface.

[0035] By performing phase unwrapping and coordinate transformation on the phase image, the three-dimensional surface geometric model of the agricultural product to be tested can be accurately reconstructed in the form of a point cloud or a triangular mesh model.

[0036] Step 202: Identification and removal of natural structures such as stems or calyxes: After obtaining the three-dimensional surface model of the agricultural product, a key application is to solve the confusion problem in traditional two-dimensional imaging.

[0037] The system can apply geometric analysis algorithms, such as calculating surface curvature, to automatically identify areas in the model with significant concave features, which typically correspond to the fruit stalk and calyx pit of agricultural products.

[0038] Once these areas are located, the system can mark them as non-defective geometric features in subsequent defect analysis, thereby ignoring them in the analysis or adopting different processing strategies. This fundamentally avoids misjudging them as defects such as bruises or rot, greatly reducing the system's false alarm rate.

[0039] Step 203, multimodal data registration and mapping: The goal of this step is to create a unified multimodal data volume. Based on the reconstructed 3D surface model, the system performs a one-time spatial calibration to establish a mapping relationship between points on the 3D model surface and each 2D imaging sensor pixel.

[0040] Then, for each agricultural product to be tested, the system will use the mapping relationship to attach the collected RGB image data, SIRI intensity data (AC and DC component images) and HSI hyperspectral data cube to the 3D model in a textured manner. The result is a highly structured data object. Figure 5 As shown in the figure, in this object, each vertex or patch on the three-dimensional model is associated with a rich set of attributes: its three-dimensional space coordinates, RGB color values, AC and DC intensity values, and a complete reflection spectrum vector.

[0041] This unified representation not only corrects the geometric distortion of the 2D image through the 3D model, but also provides a perfect, information-aligned input for subsequent fusion analysis, which is the basis for achieving high-precision fusion.

[0042] Example 3 This is one of the core technological innovations of the present invention, which is used to perform deep and intelligent feature fusion on a unified multimodal data body. Figure 2 Step S203 in .

[0043] See also Figure 3 ,The design idea of ​​CMAG-Net is to imitate the reasoning process of ,domain experts: an expert will pay more attention to the physical structure information ,when judging bruises, and will pay more attention to the color and ,chemical spectrum information when judging mold.

[0044] CMAG-Net achieves this adaptive attention allocation through its unique structure. Its network structure and workflow are as follows: the input is a local area or pixel point and its neighborhood information in the unified multimodal data volume from Example 2.

[0045] Parallel modal feature extraction branches 301, the network includes multiple parallel feature extraction branches, each branch is responsible for processing data of one modality, for example, one branch processes RGB data, one processes SIRI-AC data, one processes SIRI-DC data, and another processes HSI data after principal component analysis (PCA) dimensionality reduction; each branch is a lightweight convolutional neural network (CNN), which can be a simplified version of MobileNetV2 or ResNet, and is used to extract high-level abstract feature maps from the respective modalities.

[0046] The cross-modal attention module 302 is the key innovation of CMAG-Net. Traditional fusion methods may simply concatenate all feature maps here, but CMAG-Net adopts a more complex cross-modal attention module that draws on the self-attention concept in the Transformer architecture.

[0047] Specifically, for a certain modality, such as the feature map of SIRI-AC, the module will use it as the value and key, and the feature maps of all other modalities as queries; by calculating the dot product similarity between the query and the key, the network can learn the mutual dependence between the features of different modalities.

[0048] For example, the network may learn that when a region of the HSI feature map exhibits spectral features associated with early-stage decay, the attention weight should be increased for the corresponding region of the SIRI-AC feature map that may reflect tissue structural damage. The output of this module is a set of dynamically generated attention weight maps for each modality.

[0049] Gated fusion unit 303, original feature maps of each modality and their corresponding attention weights Figure 1 The attention weights are used as control signals to adjust the proportion of feature information of each modality flowing into the final fusion feature through the gate. If a modality is judged as unimportant or noisy by the attention module in the current context, its corresponding gate will tend to close, thereby suppressing its contribution.

[0050] After passing through the gated fusion unit, the output is that the information of all modalities is integrated into a single, highly concentrated deep fusion feature vector 304; this vector not only contains multi-source information, but more importantly, it has been intelligently filtered and weighted by the network, and its discriminative ability far exceeds that of simple feature splicing.

[0051] Example 4 This embodiment solves the bottleneck problem of insufficient rare defect samples mentioned in the background technology. Figure 2Step S204 in the above is another core technical innovation of the present invention. The basic idea is that instead of trying to generate realistic multimodal images, which is a very difficult task, it is better to directly generate high-quality feature vectors in the feature space to expand the training set.

[0052] See also Figure 4 , the detailed steps of the process are as follows: In the first step, all available rare defect samples with clear labels are collected. For example, suppose there are only 50 apple samples with bitter pit disease. These samples are passed through the preliminarily trained CMAG-Net to obtain 50 gold standard deep fusion feature vectors representing bitter pit disease. These vectors are CMAG-Net’s internal representation of this type of defect.

[0053] The second step is to build a cGAN model, which includes a generator G and a discriminator D. The input of the generator G401 is a random noise vector z and a conditional label c, which represents bitter pit disease and is usually represented in a one-hot encoding form. The task of the generator is to learn the mapping from noise to real feature vectors and output a synthesized feature vector G(z|c).

[0054] The input of the discriminator D402 is a feature vector, real or synthetic, and a conditional label c; the task of the discriminator is to determine whether the input feature vector is real under the condition of its given label, that is, whether it comes from a real rare defect sample; through adversarial training, the generator G continuously tries to generate more realistic feature vectors to deceive the discriminator D, while the discriminator D continuously improves its discrimination ability.

[0055] Eventually, when the training converges, the generator G has mastered the data distribution of this rare defect category in the feature space.

[0056] In the second step, after the training is completed, the generator G is fixed and by inputting different random noise vectors z and specified defect labels c into it, thousands of new and diverse synthetic deep fusion feature vectors 403 can be generated, but the statistical characteristics are consistent with the real samples.

[0057] In the third step, these synthesized feature vectors are combined with the real feature vectors of all other categories, including normal products and other defects, to form a large and balanced enhanced training set; this enhanced training set is used to train the final quality classifier 404, which can be a support vector machine (SVM) or a simple multi-layer perceptron (MLP). This classifier is placed on top of the CMAG-Net, corresponding to Figure 2 Step S205 in .

[0058] In this way, even if the original rare defect samples are extremely small, the final classifier can learn the robust decision boundary of the defect, thereby showing a high recall rate and precision in actual detection, i.e., step S206. This method of data augmentation in feature space rather than image space cleverly circumvents the huge complexity of multimodal image generation and provides an extremely effective and innovative solution to the problem of data imbalance in industrial inspection.

[0059] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method and system for agricultural product quality identification based on multimodal fusion and feature enhancement, characterized in that: The following steps are involved: Step S201, data acquisition: synchronously acquiring image data of at least three different modalities of the agricultural product to be tested moving along a conveyor belt using a multimodal imaging system, wherein the image data of at least three different modalities include visible light image data, structured light reflectance image data, and hyperspectral image data; Step S202, three-dimensional data mapping: reconstructing a three-dimensional surface geometric model of the agricultural product to be tested based on the phase information in the structured light reflection image data; and registering and mapping the visible light image data, the intensity information in the structured light reflection image data, and the hyperspectral image data onto the three-dimensional surface geometric model to generate a unified multimodal data volume; Step S203, feature fusion: inputting the unified multimodal data volume into a preset cross-modal attention gated fusion network, the network performs adaptive weighting processing on features of different modalities, and outputs a deep fusion feature vector; Steps S204 and S205, data augmentation: for specific defect categories with insufficient number of training samples, a preset conditional generative adversarial network is used to generate a synthetic deep fusion feature vector using the deep fusion feature vector of the specific defect category as input to expand the training dataset; Step S206, quality identification: using the augmented training data set containing the synthesized deep fusion feature vector, training a quality classifier, and using the trained quality classifier to analyze the deep fusion feature vector of the agricultural product to be tested, and finally obtaining its quality identification result.

2. The agricultural product quality identification method based on multimodal fusion and feature enhancement according to claim 1 is characterized in that: The structured light reflection image data includes a DC component image, an AC component image, and a phase image.

3. The agricultural product quality identification method based on multimodal fusion and feature enhancement according to claim 2 is characterized in that: In the three-dimensional data mapping step, the phase image is used to reconstruct a three-dimensional surface geometric model, and the model is used to identify and eliminate interference caused by natural concave areas such as stems or calyxes on the surface of agricultural products.

4. The agricultural product quality identification method based on multimodal fusion and feature enhancement according to claim 1 is characterized in that: The cross-modal attention gated fusion network includes multiple parallel modal feature extraction branches, a cross-modal attention module and a gated fusion unit; the cross-modal attention module is used to calculate the correlation weight between different modal features, and the gated fusion unit selectively fuses the modal features according to the weight.

5. The agricultural product quality identification method based on multimodal fusion and feature enhancement according to claim 4 is characterized in that: The cross-modal attention module adopts a self-attention mechanism of a Transformer-like architecture, taking the feature map of one modality as the query and the feature maps of one or more other modalities as the key and value to calculate the dependencies and attention weights between the modalities.

6. The agricultural product quality identification method based on multimodal fusion and feature enhancement according to claim 1 is characterized in that: The generator of the conditional generative adversarial network takes a random noise vector and a defect category label as conditional inputs to generate a synthetic deep fusion feature vector corresponding to the defect category; Its discriminator is used to distinguish between real deep fusion feature vectors and synthesized deep fusion feature vectors.

7. The agricultural product quality identification method based on multimodal fusion and feature enhancement according to claim 1 is characterized in that: The quality identification steps described include a hierarchical screening process: first, a rapid coarse screening is performed based on visible light image data or DC component image data with less computational effort to identify suspected defect areas; then, a complete cross-modal attention gated fusion network is called only for the suspected defect areas for refined analysis.

8. The agricultural product quality identification system based on multimodal fusion and feature enhancement is characterized by: include: A multimodal imaging module for synchronously acquiring visible light image data, structured light reflectance image data, and hyperspectral image data of the agricultural products to be tested; A processing module is connected to the multimodal imaging module, and is configured to execute the method according to any one of claims 1 to 7.

Images