Fruit and vegetable quality intelligent grading method, system and terminal

Abstract

The invention provides an intelligent grading method and system for fruit and vegetable quality and a terminal. The method comprises the following steps: firstly, acquiring multi-modal data of a fruit and vegetable object to be graded, covering an X-ray image reflecting an internal structure, an RGB image reflecting surface characteristics, and agricultural record data recording a planting environment and operation; x-ray features, visual features and farming data embedding vectors are extracted from the multi-modal data, cross-modal attention fusion is carried out on the X-ray features and the visual features on the basis of the constructed intelligent grading model, dynamic reasoning is carried out with the farming data embedding vectors as injection conditions, and a grading result is output. According to the method, by fusing the characteristics of fruit and vegetable internal maturity, wormhole detection results, surface characteristics, planting environments, fertilization records and the like, the actual measurement classification precision and efficiency are remarkably improved compared with a traditional single-mode method, the problem that surface and internal quality are not matched is reduced, multi-variety expansion is supported, and the method is suitable for large-scale popularization and application. And a more comprehensive and accurate technical scheme is provided for intelligent grading of fruits and vegetables.

Description

Technical Field

[0001] This invention relates to the field of fruit and vegetable quality testing technology, and in particular to an intelligent grading method, system and terminal for fruit and vegetable quality. Background Technology

[0002] In the fruit and vegetable industry, quality grading is a crucial step in optimizing resource utilization and enhancing commercial value, achieving rational allocation by accurately classifying fruits and vegetables of different qualities. However, current intelligent fruit and vegetable grading systems suffer from significant technical deficiencies, with insufficient fusion of multi-source heterogeneous data being the primary issue, directly leading to incomplete feature coverage. Specifically: First, surface features have extremely low correlation with intrinsic quality. For example, relying solely on visual inspection methods makes it difficult to discern key quality characteristics such as the core structure of fruits and vegetables, resulting in grading results that fail to accurately reflect their actual quality. Second, cross-variety adaptability is poor. Different fruits and vegetables possess unique physiological characteristics, while existing single models lack design flexibility and struggle to accommodate the characteristics of various fruits and vegetables, limiting the widespread application of grading systems. Third, dynamic environmental adaptability is weak. Fruit and vegetable growth is significantly affected by seasonal changes, but grading system models cannot be updated online to adapt to these changes, causing fluctuations in grading accuracy across different seasons.

[0003] Besides the aforementioned issues with data integration and adaptability, existing fruit and vegetable grading schemes also have significant shortcomings in terms of expanding to multiple varieties. Currently, most fruit and vegetable grading schemes are designed for only a single variety, lacking universality and versatility. This limitation means that in practical applications, a new grading scheme needs to be developed for each new fruit or vegetable variety introduced, increasing R&D costs and time, reducing grading efficiency, and failing to meet the needs of large-scale, diversified fruit and vegetable grading. This lack of multi-variety expansion capability severely restricts the further promotion and application of fruit and vegetable quality grading technology in the industry.

[0004] From a technological development perspective, fruit and vegetable quality grading has evolved from manual grading to mechanical grading to intelligent grading. Traditional manual grading methods are inefficient, heavily reliant on human experience, and highly subjective, making accuracy difficult to guarantee. While mechanical grading improves efficiency to some extent, it primarily relies on weight and size, failing to effectively assess the intrinsic quality of fruits and vegetables, leading to omissions and misclassifications, and potentially causing quality loss during the process. With advancements in computer and sensor technologies, intelligent grading methods have emerged, particularly those based on machine vision and spectral analysis. However, most current intelligent grading systems utilize only two-dimensional image data or single-modal data, providing limited quality information and failing to comprehensively and accurately reflect the true quality of fruits and vegetables, especially exhibiting significant shortcomings in assessing their intrinsic quality. Summary of the Invention

[0005] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a method, system and terminal for intelligent grading of fruit and vegetable quality, which can solve the technical problems of insufficient fusion of multi-source heterogeneous data, poor cross-variety adaptability, weak dynamic environment adaptability and incomplete feature coverage due to reliance on single modal data, making it difficult to accurately reflect the intrinsic quality.

[0006] To achieve the above and other related objectives, this invention provides an intelligent grading method for fruit and vegetable quality. The method includes: acquiring multimodal data for the fruit and vegetable objects to be graded; wherein the multimodal data includes: X-ray images containing the internal results of the fruit and vegetable objects, RGB images containing the surface features of the fruit and vegetable objects, and agricultural record data containing the planting environment and operation records of the fruit and vegetable objects; extracting X-ray features, visual features, and agricultural data embedding vectors based on the multimodal data of the fruit and vegetable objects to be graded; and, based on the constructed intelligent grading model for fruit and vegetable quality, fusing the extracted X-ray features and visual features through cross-modal attention, and using the agricultural data embedding vectors as injection conditions to perform dynamic inference and output the corresponding fruit and vegetable quality grading results.

[0007] In one embodiment of the present invention, the extraction of X-ray features, visual features, and agricultural data embedding vectors based on multimodal data of fruits and vegetables to be graded includes: synchronizing X-ray images, RGB images, and agricultural record data of fruits and vegetables through timestamps, and performing data augmentation on the synchronized X-ray images, RGB images, and agricultural record data; dividing the data-augmented X-ray images and RGB images into blocks, and converting them into embedding vectors using linear projection layers to obtain X-ray features and visual features; encoding the data-augmented agricultural record data into multidimensional vectors, and mapping them into agricultural data embedding vectors through fully connected layers.

[0008] In one embodiment of the present invention, the intelligent fruit and vegetable quality grading model includes: a feature fusion network, used to fuse input X-ray features and visual features into bimodal fusion features through cross-modal attention using a weighted gating mechanism; a main feature extraction network, connected to the feature fusion network, used to extract multi-scale features based on the bimodal fusion features and dynamically inject agricultural data embedding vectors to generate feature maps; and a quality grading network, connected to the main feature extraction network, used to output corresponding fruit and vegetable quality grading results based on the feature maps.

[0009] In one embodiment of the present invention, the main feature extraction network adopts an architecture of alternating stacking of feature extraction modules and SpatialTransformer layers.

[0010] In one embodiment of the present invention, the main feature extraction network includes: a four-level feature extraction module and a four-level SpatialTransformer layer; wherein, a SpatialTransformer layer is inserted after each feature extraction module; each feature extraction module is used to downsample the input features and calculate the residuals to extract multi-scale features; each SpatialTransformer layer is used to fuse the multi-scale features output by the previous feature extraction module with the injected agricultural data embedding vector through a cross-attention mechanism, until the fourth-level SpatialTransformer layer outputs the final feature map.

[0011] In one embodiment of the present invention, each feature extraction module includes: a downsampling layer, used to multiply the number of channels of the input features through a 3×3 convolution operation with a stride of 2; two stacked residual blocks; each residual block is used to extract features through a 3×3 convolution layer connected to a SiLU activation function, then normalized by GroupNorm, and then fused with the time-step embedding information through MLP processing and features, and finally the original input and the processed features are added through residual connections to extract multi-scale features.

[0012] In one embodiment of the present invention, the quality grading network includes: a global average pooling layer and a three-level classifier connected to the global average pooling layer and deployed thereon; the feature map output by the main feature extraction network is first compressed into channel vectors by the global average pooling layer, and then multi-level probability distributions are generated by the deployed three-level classifier containing fully connected layers, and joint optimization is performed by weighted cross-entropy loss to output results with different fine granularities.

[0013] In one embodiment of the present invention, the three-level classifier includes: a first-level classifier, used to classify fruits and vegetables into qualified or unqualified categories based on the degree of epidermal damage and internal rot, generating a first-level classification result; a second-level classifier, used to classify fruits and vegetables into different quality levels according to sugar content distribution, size, and weight based on the first-level classification result, generating a second-level classification result; and a third-level classifier, used to label fruits and vegetables based on the second-level classification result, generating a third-level classification result.

[0014] To achieve the above and other related objectives, this invention provides an intelligent grading system for fruit and vegetable quality. The system includes: a multimodal data acquisition module for acquiring multimodal data for the fruits and vegetables to be graded; wherein the multimodal data includes: X-ray images containing internal characteristics of the fruits and vegetables, RGB images containing surface features of the fruits and vegetables, and agricultural record data containing the planting environment and operational records of the fruits and vegetables; a multimodal feature extraction module connected to the multimodal data acquisition module for extracting X-ray features, visual features, and agricultural data embedding vectors based on the multimodal data of the fruits and vegetables to be graded; and a quality grading module connected to the multimodal feature extraction module for dynamically inferring the extracted X-ray features and visual features through cross-modal attention fusion based on a constructed intelligent grading model for fruit and vegetable quality, and outputting the corresponding fruit and vegetable quality grading result by using the agricultural data embedding vector as an injection condition.

[0015] To achieve the above and other related objectives, the present invention provides an electronic terminal, comprising: one or more memories and one or more processors; the one or more memories are used to store a computer program; the one or more processors are connected to the memories and are used to run the computer program to execute the intelligent grading method for fruit and vegetable quality.

[0016] As described above, this invention provides a method, system, and terminal for intelligent grading of fruit and vegetable quality, offering the following advantages: First, the invention acquires multimodal data of the fruits and vegetables to be graded, including X-ray images reflecting internal structure, RGB images showcasing surface features, and agricultural records documenting the planting environment and operations. Then, it extracts X-ray features, visual features, and agricultural data embedding vectors from the multimodal data. Based on the constructed intelligent grading model, it performs cross-modal attention fusion of X-ray and visual features, uses the agricultural data embedding vectors as injection conditions for dynamic inference, and outputs the grading results. By integrating features such as internal maturity, wormhole detection results, surface features, planting environment, and fertilization records, this invention significantly improves the accuracy and efficiency of classification compared to traditional single-modal methods, reduces the mismatch between surface and internal quality, and supports multi-variety expansion, providing a more comprehensive and accurate technical solution for intelligent grading of fruits and vegetables. Attached Figure Description

[0017] Figure 1 The diagram shown is a flowchart of an intelligent fruit and vegetable quality grading method according to an embodiment of the present invention.

[0018] Figure 2 The diagram shown is a structural schematic of an intelligent fruit and vegetable quality grading model according to an embodiment of the present invention.

[0019] Figure 3 The diagram shown is a schematic of the intelligent grading process for fruit and vegetable quality according to an embodiment of the present invention.

[0020] Figure 4 The diagram shown is a structural schematic of an intelligent fruit and vegetable quality grading system according to an embodiment of the present invention.

[0021] Figure 5 The diagram shown is a structural schematic of an electronic terminal according to an embodiment of the present invention. Detailed Implementation

[0022] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0023] It should be noted that in the following description, reference is made to the accompanying drawings, which illustrate several embodiments of the invention. It should be understood that other embodiments may also be used, and changes in mechanical composition, structure, electrical system, and operation may be made without departing from the spirit and scope of the invention. The following detailed description should not be considered limiting, and the scope of the embodiments of the invention is defined only by the claims of the published patents. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. Spatially related terms, such as “upper,” “lower,” “left,” “right,” “below,” “below,” “lower part,” “above,” “upper part,” etc., may be used herein to illustrate the relationship between one element or feature shown in the figures and another element or feature.

[0024] Throughout this specification, when it is said that a part is "connected" to another part, this includes not only "direct connection" but also "indirect connection" by placing other elements in between. Furthermore, when it is said that a part "includes" a certain constituent element, unless otherwise stated otherwise, this does not exclude other constituent elements, but rather means that other constituent elements may also be included.

[0025] The terms "first," "second," and "third," etc., used herein are for the purpose of describing various parts, components, regions, layers, and / or segments, but are not limiting. These terms are used only to distinguish one part, component, region, layer, or segment from others. Therefore, the "first part," "component," "region," "layer," or "segment" described below may refer to a "second part," "component," "region," "layer," or "segment" without departing from the scope of this invention.

[0026] Furthermore, as used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context indicates otherwise. It should be further understood that the terms “comprising,” “including,” indicate the presence of the stated feature, operation, element, component, item, kind, and / or group, but do not preclude the presence, occurrence, or addition of one or more other features, operations, elements, components, items, kinds, and / or groups. The terms “or” and “and / or” as used herein are interpreted as inclusive, or mean any one or any combination thereof. Thus, “A, B, or C” or “A, B, and / or C” means “any one of: A; B; C; A and B; A and C; B and C; A, B, and C.” Exceptions to this definition arise only when combinations of elements, functions, or operations are inherently mutually exclusive in some manner.

[0027] This invention provides an intelligent grading method for fruit and vegetable quality. It first acquires multimodal data of the fruits and vegetables to be graded, including X-ray images reflecting internal structure, RGB images showing surface features, and agricultural records documenting the planting environment and operations. Then, it extracts X-ray features, visual features, and agricultural data embedding vectors from the multimodal data. Based on a constructed intelligent grading model, it performs cross-modal attention fusion of X-ray and visual features, uses the agricultural data embedding vectors as injection conditions for dynamic inference, and outputs the grading results. This invention, by integrating features such as internal maturity, insect hole detection results, surface features, planting environment, and fertilization records, significantly improves the classification accuracy and efficiency compared to traditional single-modal methods, reduces the mismatch between surface and internal quality, and supports multi-variety expansion, providing a more comprehensive and accurate technical solution for intelligent fruit and vegetable grading.

[0028] The present invention will now be described in detail with reference to the accompanying drawings, so that those skilled in the art can readily implement it. The present invention can be embodied in many different forms and is not limited to the embodiments described herein.

[0029] like Figure 1 This is a flowchart illustrating an intelligent grading method for fruit and vegetable quality according to an embodiment of the present invention.

[0030] The method includes:

[0031] Step S1: Collect multimodal data for the fruits and vegetables to be graded.

[0032] In detail, the multimodal data includes: X-ray images containing the internal results of fruits and vegetables, RGB images containing the surface features of fruits and vegetables, and agricultural record data containing the planting environment and operation records of fruits and vegetables;

[0033] The X-ray imaging module acquires X-ray images of fruits and vegetables, detecting invisible internal quality defects such as core distribution, wormholes, and internal damage. X-ray images provide penetrating imaging, compensating for the "blind spots" of visual inspection regarding internal defects. The module employs low-dose scanning (0.1–0.5 mGy), meeting food safety standards (international food irradiation dose ≤10 kGy), ensuring the safety of fruits and vegetables for consumption. With a spatial resolution ≥200 μm, it can identify millimeter-scale internal defects (such as wormholes with a diameter >0.2 mm).

[0034] The agricultural data module collects agricultural records of fruits and vegetables; it also collects environmental data throughout the entire planting cycle to establish the correlation between growth conditions and quality. Specific data includes: environmental parameters such as temperature and humidity (affecting sugar accumulation), light duration (affecting pigment synthesis), and rainfall (affecting fruit water content); and agricultural operation parameters such as fertilizer type (e.g., nitrogen fertilizer affects fruit tenderness), fertilizer amount, and harvest time (affecting maturity). The agricultural record data dynamically tracks the growth environment, solving the traceability problem of "quality differences between the same variety from different production areas."

[0035] The vision module acquires RGB images; these images contain visible features of the fruit and vegetable surface, enabling rapid screening of appearance quality. Specifically, these include: color features for analyzing ripeness; texture features for identifying minor defects such as abrasions, mold, and rust; and shape features for detecting deformed fruit. The vision module can utilize a 20-megapixel CMOS sensor and is equipped with a multispectral LED ring light source (wavelength range 400-1000nm) to acquire RGB images.

[0036] This solution's collection of multimodal data is of great significance: X-ray images can penetrate and detect the internal quality of fruits and vegetables, such as the distribution of sugar cores, wormholes, and internal damage, compensating for the "blind spots" of traditional visual inspection regarding internal defects; RGB images can extract surface color, texture, and shape features, enabling rapid screening of appearance quality, but cannot reveal internal conditions; agricultural data integrates information such as planting environment and operation records, establishing a correlation between growth conditions and quality, and solving the traceability problem of quality differences among the same variety from different production areas. With these three technologies working together, it can simultaneously identify internal and external defects in fruits and vegetables, avoiding discrepancies between appearance and reality, and improve grading accuracy through environmental and quality correlation analysis, significantly reducing the false negative rate and achieving a comprehensive and precise assessment of fruit and vegetable quality.

[0037] Step S2: Extract X-ray features, visual features, and agricultural data embedding vectors from the multimodal data of the fruits and vegetables to be graded.

[0038] In one embodiment, step S2 includes:

[0039] By synchronizing X-ray images, RGB images, and agricultural records of fruits and vegetables using timestamps, it is ensured that the three types of data correspond to the same growth stage of the fruit or vegetable.

[0040] Data augmentation was performed on the synchronized X-ray images, RGB images, and agricultural record data. Specifically, X-ray image data augmentation methods included: using median filtering and Gaussian denoising algorithms to eliminate scanning noise while preserving internal structural details (such as sugar core boundaries and wormhole outlines). RGB image data augmentation methods included: applying random cropping and random lighting adjustments to enhance the model's robustness to different shooting angles and lighting conditions. Agricultural record data augmentation methods included: normalizing continuous data such as temperature, humidity, and light duration, and using one-hot encoding for discrete data such as fertilizer type to improve data distribution stability.

[0041] The augmented X-ray and RGB images are segmented into blocks and converted into embedding vectors using a linear projection layer to obtain X-ray and visual features. Specifically, the X-ray and RGB images are divided into fixed-size sub-blocks (e.g., 16×16 pixels) to facilitate subsequent feature extraction. For each image sub-block, the pixel values ​​are converted into low-dimensional embedding vectors using a linear projection layer. The output X-ray and visual features are feature matrices consisting of the number of sub-blocks multiplied by the embedding dimension.

[0042] The data-enhanced agricultural records are encoded into multidimensional vectors, which are then mapped to agricultural data embedding vectors through a fully connected layer. Specifically, the structured agricultural data is first encoded into multidimensional vectors. A non-linear mapping is then performed through a fully connected layer to generate agricultural data embedding vectors with dimensions consistent with the image features, ensuring dimensionality compatibility of multimodal features during subsequent fusion.

[0043] This invention addresses the heterogeneity of X-ray, visual, and agricultural data in terms of spatiotemporal and feature spaces by synchronizing and unifying the timestamps and dimensions of multimodal data. Data augmentation techniques effectively expand the diversity of training data, reduce the risk of overfitting, and particularly enhance the model's ability to detect rare defects and complex environments. Block processing allows the model to capture local features, which, combined with global environmental information embedded in agricultural data, provide a fine-grained feature foundation for subsequent cross-modal attention fusion.

[0044] Step S3: Based on the constructed intelligent grading model for fruit and vegetable quality, the extracted X-ray features and visual features are fused through cross-modal attention, and the embedded vector of agricultural data is used as the injection condition to perform dynamic reasoning and output the corresponding fruit and vegetable quality grading results.

[0045] In one embodiment, such as Figure 2The intelligent grading model for fruit and vegetable quality adopts the AG-ViT (agricultureVision Transformer) algorithm, which is a deep multimodal Transformer model specifically designed for agricultural product grading, including:

[0046] The feature fusion network is used to fuse input X-ray features and visual features into bimodal fusion features through cross-modal attention using a weighted gating mechanism. Specifically, a gating unit is introduced to dynamically adjust the fusion ratio of bimodal features and fuse them into bimodal fusion features through cross-modal attention.

[0047] The main feature extraction network, connected to the feature fusion network, is used to extract multi-scale features based on dual-modal fusion features and dynamically inject agricultural data embedding vectors to generate feature maps.

[0048] A quality grading network, connected to the main feature extraction network, is used to output corresponding fruit and vegetable quality grading results based on the feature map.

[0049] In one embodiment, the main feature extraction network employs an architecture that alternately stacks feature extraction modules and SpatialTransformer layers.

[0050] In one embodiment, such as Figure 3 The main feature extraction network consists of four levels of feature extraction modules and four levels of SpatialTransformer layers, with each feature extraction module connected to a SpatialTransformer layer, forming a cascaded structure of alternating modules and layers. This cascaded design enables a hierarchical feature extraction process from low-level local features to high-level global semantics. Furthermore, it can dynamically adjust based on agricultural data during spatial transformations to better adapt to the feature extraction needs of different agricultural scenarios, providing accurate and comprehensive feature support for subsequent fruit and vegetable quality grading tasks.

[0051] The input to the first-level feature extraction module is the dual-modal fused features output by the feature fusion network; the input to the feature extraction modules from the second to the fourth level is the feature map output by the previous level SpatialTransformer layer.

[0052] Each feature extraction module undertakes a key task: it downsamples the input features and performs residual calculations to extract multi-scale feature information, providing a rich feature foundation for subsequent processing.

[0053] Each SpatialTransformer layer is a crucial step in the fusion process. It receives multi-scale features from the previous feature extraction module, as well as dynamically injected agricultural data embedding vectors. These agricultural data embedding vectors serve as conditional control signals for the SpatialTransformer, enabling the network to selectively fuse and transform features based on different agricultural data conditions. A cross-attention mechanism is employed for deep fusion. This mechanism allows for effective information interaction and mutual influence between features from different sources, resulting in fused features that better integrate multi-scale features and agricultural data information, leading to more representative and discriminative feature representations. This fusion process is progressive, with each level further optimizing and improving the quality and semantic level of the features based on the previous level. After four layers of SpatialTransformer fusion and transformation, the final feature map is output by the fourth-level SpatialTransformer layer. This final feature map contains rich, dynamically adjusted, and deeply fused feature information, accurately and comprehensively reflecting the quality characteristics of fruits and vegetables. It provides a reliable feature basis for subsequent fruit and vegetable quality grading tasks, contributing to improved accuracy and reliability in grading.

[0054] In one embodiment, such as Figure 3 Each feature extraction module includes:

[0055] The downsampling layer is used to downsample the input features while increasing the number of channels in the features. Specifically, the downsampling layer uses a 3×3 convolution operation with a stride of 2. This convolution operation can multiply the number of channels in the input features while reducing the dimensionality of the feature space (i.e., downsampling). This helps to increase the expressive power of the features while reducing the spatial resolution, providing richer channel information for subsequent feature extraction.

[0056] Two stacked residual blocks (ResBlocks) are used; each residual block is used to extract multi-scale features and avoids the gradient vanishing problem in deep networks through residual connections, ensuring that the network can be trained effectively. Each residual block first convolves the input features through a 3×3 convolutional layer to extract new features. Then, the convolutional features are non-linearly transformed by the SiLU (Sigmoid Linear Unit) activation function. The SiLU activation function has self-gating properties, which can dynamically adjust the output according to the input, helping the model learn more complex feature representations. The features after the activation function are normalized by GroupNorm. GroupNorm is a normalization method that divides the feature channels into several groups and normalizes the features within each group. Next, the temporal step embedding information is processed by an MLP (Multilayer Perceptron). The temporal step embedding information may be time-related features or signals, used to provide the model with temporal dimension information. The MLP can learn and transform this temporal step embedding information, enabling it to be effectively fused with the features. The temporal embedding information processed by the MLP is fused with the features after convolution, activation, and normalization. This fusion method incorporates temporal information into spatial features, making the features richer and more comprehensive. Finally, residual connections are used to add the original input features to the processed features, extracting multi-scale features. The role of residual connections is to directly pass input features to the output, avoiding the information loss problem in deep networks, helping the model learn more complex feature representations, and maintaining network stability.

[0057] Therefore, it can be seen that the multi-scale feature extraction capability of the main feature extraction network can comprehensively capture the details and global features of fruit and vegetable quality, accurately extracting both local defects and overall shape and color, providing rich information for grading. Secondly, by integrating time-dimensional information, the model can deeply understand the quality changes of fruits and vegetables at different time stages, such as the impact of storage time on the freshness of strawberries, thereby improving grading accuracy. In addition, the SpatialTransformer layer dynamically adjusts the spatial transformation based on agricultural data, enabling the model to flexibly adapt to complex scenarios such as different fruit and vegetable varieties, growing environments, and harvesting conditions, enhancing grading adaptability. At the same time, the deep residual network structure effectively avoids gradient problems, helping the model learn complex features and accurately distinguish subtle differences between different quality grades. Finally, the cascaded design and feature fusion operation within the residual block achieve full complementarity of features from different sources and levels, and the comprehensive feature representation obtained by integrating multiple factors further improves the accuracy and reliability of fruit and vegetable quality grading.

[0058] In one embodiment, such as Figure 3 The quality grading network includes:

[0059] The global average pooling layer receives the feature map output by the main feature extraction network. Its function is to perform average pooling on the feature map along its spatial dimensions (height and width), compressing the spatial features of each channel into a single numerical value, resulting in a channel vector. This channel vector retains the important information of the feature map in each channel while eliminating spatial differences, providing a compact feature representation for the subsequent classifier.

[0060] The network also deploys a three-level classifier; the output of the global average pooling layer is connected to the deployed three-level classifier, each classifier is responsible for different fine-grained classification tasks.

[0061] The compressed channel vectors are input into the deployed three-level classifiers. Each classifier maps the channel vectors to the corresponding class probability distributions through its internal fully connected layers. To train this multi-level classification network, a weighted cross-entropy loss function is used for joint optimization. Specifically, the loss weight for level one classification is 0.3, for level two classification it is 0.4, and for level three classification it is 0.3. This weighting method allows the model to simultaneously optimize classification tasks of different granularities during training, ensuring that each classifier effectively learns the corresponding classification features while balancing the importance of different classification tasks.

[0062] In practical applications, the grading thresholds can be dynamically adjusted based on industry standards and specific needs to support custom grading. For example, specific grading criteria for each quality level can be flexibly defined according to market requirements or quality standards, making the grading results more consistent with the needs of actual application scenarios.

[0063] In one specific embodiment, such as Figure 3 The deployed three-level classifier includes:

[0064] The primary classifier is responsible for coarse-grained quality classification of fruits and vegetables, outputting a basic quality level of "qualified" or "unqualified". Specifically, it classifies fruits and vegetables into qualified or unqualified categories based on the degree of skin damage and internal rot, and can output the results. This is a preliminary judgment of the overall quality of fruits and vegetables, providing a foundation for subsequent more detailed grading.

[0065] The secondary classifier, building upon the primary classification, further refines the division of each basic quality level. Specifically, under the "qualified" category, it extracts key indicators such as sugar content range, size range, and weight range, classifying qualified fruits and vegetables into different quality levels based on sugar content distribution, size, and weight. This allows for a more precise description of the quality characteristics of fruits and vegetables, meeting the specific needs of different application scenarios.

[0066] The three-level classifier uses the results of the two-level classification to label information such as variety and place of origin in agricultural characteristics. The label includes: specific place of origin (such as "Gannan Navel Orange" or "Fengjie Navel Orange"), variety, quality grade (high-quality fruit / qualified fruit), production date and other information.

[0067] These three classifiers can be flexibly combined according to different needs: the primary classifier can be used alone to quickly output the basic quality grade of "qualified" or "unqualified", which is suitable for large-scale initial screening to remove obviously unqualified products; it can also be linked with the primary and secondary classifiers to further refine the indicators such as sugar content, size, and weight of qualified fruits and vegetables based on the initial screening, to meet the needs of fine grading such as pricing by quality and matching with processing technology; it can also realize the full-process linkage of the three classifiers, and while outputting the basic quality grade and refined indicators, it can also superimpose traceability labels such as place of origin, variety, and production date, which is suitable for scenarios with high requirements for product added value and traceability, such as brand building, high-end market sales, and full-chain traceability, thereby realizing the output of different granular classification results from basic sorting to fine grading and then to brand empowerment.

[0068] Through the progressive refinement of the three-level classifier, the final grading results not only include basic quality grades (such as "qualified" and "unqualified"), but also cover more specific indicators, such as sugar content range, weight range, and even further refinement to information such as origin. This fine-grained grading result can more comprehensively reflect the quality characteristics of fruits and vegetables, providing more detailed reference for subsequent sales, processing, and storage.

[0069] In one embodiment, the method for constructing the intelligent grading model for fruit and vegetable quality includes:

[0070] First, prepare the dataset for training the model, specifically including: collecting multimodal data on fruits and vegetables of various varieties and quality levels. This data includes: X-ray images containing the internal structure of the fruits and vegetables, RGB images containing surface features, and agricultural records containing the planting environment and operational records. Synchronize the X-ray images, RGB images, and agricultural records for each fruit and vegetable object using timestamps. Ensure that these different modalities are consistent in time to enable accurate correlation and analysis. Perform data augmentation on the synchronized X-ray images, RGB images, and agricultural records. Extract X-ray features, visual features, and agricultural data embedding vectors from the augmented X-ray images, RGB images, and agricultural records. Combine the X-ray features, visual features, and agricultural data embedding vectors of each fruit and vegetable object into a sample and label it with the corresponding variety and quality level. These labels can be predefined classification labels, such as "high-quality," "qualified," and "unqualified." Divide the constructed training dataset into training, validation, and test sets to facilitate effective evaluation and adjustment during model training. A smart grading model for fruit and vegetable quality is trained using a training dataset. During training, techniques such as cross-validation can be used to evaluate the model's performance, and the model's parameters can be adjusted based on feedback from the validation set. The trained model is then evaluated using a test set, and further optimizations are made based on the evaluation results, such as adjusting the network structure, adding regularization terms, and adjusting the learning rate, to improve the model's generalization ability and performance.

[0071] In one embodiment, by deploying models on edge computing devices (such as Jetson AGX), real-time sorting at the production line level can be achieved, with a processing speed of over 50 items per second, significantly improving sorting efficiency and meeting the high-speed processing requirements of industrial production lines. Simultaneously, this automated sorting method drastically reduces manual sorting costs, saving over 60% of labor costs, bringing significant economic benefits to enterprises and improving production efficiency and product quality consistency.

[0072] To better describe the above-mentioned intelligent grading method for fruit and vegetable quality, the following specific embodiments are provided for illustration.

[0073] Example 1: A method for intelligent grading of orange quality.

[0074] Step 1: Acquire multimodal data of the oranges to be graded by collecting data from the X-ray module, vision module, and agricultural data module.

[0075] An X-ray module scans the internal structure of oranges to detect the distribution of the core, internal rot, or wormholes. A vision module captures surface color, texture, and shape. An agricultural data module connects to a planting database to obtain environmental parameters of the orange's production area (such as sunshine duration and rainfall), harvest time, and fertilization records, which are used to help determine sugar content and storage tolerance. Multimodal data is synchronized via timestamps to ensure a one-to-one correspondence between X-ray images, photographs, and agricultural information for the same orange.

[0076] Step 2: Perform multimodal fusion and dynamic inference on the multimodal data of oranges.

[0077] Employing an advanced AG-ViT model, this system deeply integrates X-ray features (reflecting internal quality) and visual features (reflecting surface quality) through a cross-modal attention mechanism, and cleverly incorporates agricultural conditions (e.g., "oranges from high-rainfall areas require special attention to sugar content") to accurately generate a comprehensive quality score, providing comprehensive and accurate data support for grading decisions. The system can dynamically update parameters based on real-time sorting data through an incremental learning mechanism, quickly adapting to quality fluctuations in different batches of oranges (such as lower sugar content during the rainy season), ensuring the accuracy and timeliness of grading results, and effectively coping with complex and ever-changing production environments.

[0078] Step 3: Grade the oranges according to their quality.

[0079] First, a primary classification process quickly identifies "unqualified" oranges (approximately 5%) with severely damaged peels or internal rot, strictly controlling quality to ensure subsequent oranges meet basic quality requirements. Next, a secondary classification categorizes oranges based on precise sugar content (above 12°Brix is ​​considered premium) and weight (200-300g is the standard). Premium oranges are directly supplied to the high-end market to meet high-quality demands, while qualified oranges are used for juice processing, achieving rational resource utilization and maximizing product added value. Finally, a tertiary classification process meticulously labels the origin (e.g., "Gannan Navel Orange"), facilitating branded packaging, enhancing product market competitiveness, meeting consumer demand for regional characteristics, and promoting brand building and development.

[0080] Step 4: Provide real-time feedback on the quality grading results of the oranges:

[0081] The results of fruit and vegetable quality grading are synchronized to the MES (Manufacturing Execution System) in real time, providing accurate data support for harvesting and logistics scheduling, effectively optimizing the production process, reducing unsold losses, and improving overall operational efficiency.

[0082] This embodiment achieves fully automated sorting, processing up to 180,000 oranges per hour with a false negative rate of less than 0.5%, representing an efficiency improvement of more than 10 times compared to traditional manual sorting methods. Utilizing multimodal data fusion technology, it can accurately identify high-value-added varieties such as "sugar heart oranges," significantly increasing the premium sales ratio by up to 25%. Simultaneously, the dynamic optimization model can flexibly adapt to the needs of different production areas, such as the differentiated configuration for grading standards of South African navel oranges and domestic Gannan navel oranges, providing strong technical support for global supply chain deployment and helping enterprises gain a dominant position in the global market.

[0083] Similar to the principles of the above embodiments, the present invention provides a method for intelligent grading of fruit and vegetable quality.

[0084] The following specific embodiments are provided in conjunction with the accompanying drawings:

[0085] like Figure 4 This invention presents a schematic diagram of the structure of an intelligent grading system for fruit and vegetable quality according to an embodiment of the present invention.

[0086] The system includes:

[0087] The multimodal data acquisition module 1 is used to acquire multimodal data for the fruits and vegetables to be graded; wherein, the multimodal data includes: X-ray images containing the internal results of the fruits and vegetables, RGB images containing the surface features of the fruits and vegetables, and agricultural record data containing the planting environment and operation records of the fruits and vegetables.

[0088] The multimodal feature extraction module 2 is connected to the multimodal data acquisition module 1 and is used to extract X-ray features, visual features, and agricultural data embedding vectors based on the multimodal data of the fruits and vegetables to be graded.

[0089] The quality grading module 3 is connected to the multimodal feature extraction module 2. It is used to extract X-ray features and visual features based on the constructed intelligent fruit and vegetable quality grading model, fuse them through cross-modal attention, and use the agricultural data embedding vector as the injection condition to perform dynamic reasoning and output the corresponding fruit and vegetable quality grading results.

[0090] Since the implementation principle of the intelligent grading system for fruit and vegetable quality has been described in the foregoing embodiments, it will not be repeated here.

[0091] The intelligent fruit and vegetable quality grading method provided in this invention can be implemented on the terminal side or the server side. Regarding the hardware structure of the electronic terminal, please refer to [link to relevant documentation]. Figure 5This is a schematic diagram of an optional hardware structure of an electronic terminal 1000 provided in an embodiment of the present invention. The terminal 1000 can be a mobile phone, computer device, tablet device, personal digital processing device, factory back-end processing device, etc. The terminal 1000 includes: at least one processor 1001, a memory 1002, at least one network interface 10010, and a user interface 1009. The various components in the device are coupled together through a bus system 1005. It is understood that the bus system 1005 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 1005 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 5 The general will label all buses as bus systems.

[0092] The user interface 1009 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0093] It is understood that memory 1002 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0094] In this embodiment of the invention, the memory 1002 is used to store various types of data to support the operation of the terminal 1000. Examples of this data include: any executable program for operation on the terminal 1000, such as the operating system 10021 and application program 10022; the operating system 10021 includes various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 10022 may include various applications, such as a media player, browser, etc., for implementing various application services. The intelligent fruit and vegetable quality grading method provided in this embodiment of the invention can be included in the application program 10022.

[0095] The methods disclosed in the above embodiments of the present invention can be applied to or implemented by the processor 1001. The processor 1001 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor 1001 or by instructions in the form of software. The processor 1001 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 1001 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor 1001 may be a microprocessor or any conventional processor, etc. The steps of the accessory optimization method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in a memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0096] In an exemplary embodiment, the terminal 1000 may be used to execute the aforementioned method by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs).

[0097] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented using computer program-related hardware. The aforementioned computer program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0098] In the embodiments provided in this application, the computer-readable and writable storage medium may include read-only memory, random access memory, EEPROM, CD-ROM or other optical disc storage devices, disk storage devices or other magnetic storage devices, flash memory, USB flash drive, portable hard drive, or any other medium capable of storing desired program code in the form of instructions or data structures and accessible by a computer. Additionally, any connection may be appropriately referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. However, it should be understood that computer-readable and writable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are intended for non-transient, tangible storage media. The disks and optical discs used in the application include compact discs (CDs), laser discs, optical discs, digital multifunction discs (DVDs), floppy disks, and Blu-ray discs, where disks typically copy data magnetically, while optical discs use lasers to copy data optically.

[0099] In summary, the intelligent fruit and vegetable quality grading method, system, and terminal of this invention first acquire multimodal data of the fruits and vegetables to be graded, including X-ray images reflecting internal structure, RGB images showing surface features, and agricultural records documenting the planting environment and operations. Then, X-ray features, visual features, and agricultural data embedding vectors are extracted from the multimodal data. Based on the constructed intelligent grading model, cross-modal attention fusion of X-ray and visual features is performed, and dynamic inference is performed using the agricultural data embedding vectors as injection conditions to output the grading results. This invention, by integrating features such as internal maturity of fruits and vegetables, wormhole detection results, surface features, planting environment, and fertilization records, significantly improves the measured classification accuracy and efficiency compared to traditional single-modal methods, reduces the mismatch between surface and internal quality, and supports multi-variety expansion, providing a more comprehensive and accurate technical solution for intelligent fruit and vegetable grading. Therefore, this invention effectively overcomes the various shortcomings of existing technologies and has high industrial application value.

[0100] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A method for intelligent grading of fruit and vegetable quality, characterized in that, The method includes: Multimodal data is collected for the fruits and vegetables to be graded; wherein, the multimodal data includes: X-ray images containing the internal results of the fruits and vegetables, RGB images containing the surface features of the fruits and vegetables, and agricultural record data containing the planting environment and operation records of the fruits and vegetables. X-ray features, visual features, and agricultural data embedding vectors are extracted from multimodal data of fruits and vegetables to be graded. Based on the constructed intelligent grading model for fruit and vegetable quality, X-ray features and visual features are extracted and fused through cross-modal attention. Agricultural data embedding vectors are used as injection conditions to dynamically infer and output the corresponding fruit and vegetable quality grading results.

2. The intelligent grading method for fruit and vegetable quality according to claim 1, characterized in that, The extraction of X-ray features, visual features, and agricultural data embedding vectors from multimodal data of fruits and vegetables to be graded includes: The X-ray images, RGB images, and agricultural records of fruits and vegetables are synchronized using timestamps, and data augmentation is performed on the synchronized X-ray images, RGB images, and agricultural records. The augmented X-ray and RGB images are divided into blocks and converted into embedding vectors using a linear projection layer to obtain X-ray features and visual features. The data-enhanced agricultural records are encoded into multidimensional vectors and mapped to agricultural data embedding vectors through a fully connected layer.

3. The intelligent grading method for fruit and vegetable quality according to claim 1, characterized in that, The intelligent grading model for fruit and vegetable quality includes: The feature fusion network is used to fuse input X-ray features and visual features into bimodal fusion features through cross-modal attention using a weighted gating mechanism. The main feature extraction network, connected to the feature fusion network, is used to extract multi-scale features based on dual-modal fusion features and dynamically inject agricultural data embedding vectors to generate feature maps. A quality grading network, connected to the main feature extraction network, is used to output corresponding fruit and vegetable quality grading results based on the feature map.

4. The intelligent grading method for fruit and vegetable quality according to claim 3, characterized in that, The main feature extraction network adopts an architecture that alternately stacks feature extraction modules and SpatialTransformer layers.

5. The intelligent grading method for fruit and vegetable quality according to claim 4, characterized in that, The main feature extraction network includes: a four-level feature extraction module and a four-level SpatialTransformer layer; wherein, a SpatialTransformer layer is inserted after each feature extraction module. Each feature extraction module is used to downsample the input features and calculate residuals to extract multi-scale features; Each SpatialTransformer layer is used to fuse the multi-scale features output by the previous feature extraction module with the injected agricultural data embedding vector through a cross-attention mechanism, until the fourth SpatialTransformer layer outputs the final feature map.

6. The intelligent grading method for fruit and vegetable quality according to claim 5, characterized in that, Each feature extraction module includes: The downsampling layer is used to multiply the number of channels of the input features through a 3×3 convolution operation with a stride of 2; Two stacked residual blocks are used. Each residual block is used to extract features through a 3×3 convolutional layer connected to the SiLU activation function, then normalized by GroupNorm, and then processed by MLP to embed information of time steps and fuse with features. Finally, the original input and the processed features are added through residual connections to extract multi-scale features.

7. The intelligent grading method for fruit and vegetable quality according to claim 1, characterized in that, The quality grading network includes: a global average pooling layer and a three-level classifier connected to and deployed on the global average pooling layer; The feature map output by the main feature extraction network is first compressed into channel vectors by a global average pooling layer, and then multi-level probability distributions are generated by a three-level classifier with fully connected layers. Weighted cross-entropy loss is used for joint optimization to output results with different fine granularities.

8. The intelligent grading method for fruit and vegetable quality according to claim 7, characterized in that, The three-level classifier includes: A primary classifier is used to classify fruits and vegetables into qualified or unqualified categories based on the degree of skin damage and internal decay, and to generate primary classification results. A secondary classifier is used to classify fruits and vegetables into different quality levels based on the primary classification results, according to sugar content distribution, size, and weight, and generate secondary classification results. A tertiary classifier is used to label fruit and vegetable objects based on the secondary classification results and generate tertiary classification results.

9. A smart grading system for fruit and vegetable quality, characterized in that, The system includes: The multimodal data acquisition module is used to acquire multimodal data for the fruits and vegetables to be graded; wherein, the multimodal data includes: X-ray images containing the internal results of the fruits and vegetables, RGB images containing the surface features of the fruits and vegetables, and agricultural record data containing the planting environment and operation records of the fruits and vegetables. The multimodal feature extraction module, connected to the multimodal data acquisition module, is used to extract X-ray features, visual features, and agricultural data embedding vectors based on the multimodal data of the fruits and vegetables to be graded. The quality grading module, connected to the multimodal feature extraction module, is used to extract X-ray features and visual features based on the constructed intelligent fruit and vegetable quality grading model. Through cross-modal attention fusion, it uses agricultural data embedding vectors as injection conditions to perform dynamic reasoning and output the corresponding fruit and vegetable quality grading results.

10. An electronic terminal, characterized in that, include: One or more memories and one or more processors; The one or more memories are used to store computer programs; The one or more processors are connected to the memory and are used to run the computer program to perform the method as described in any one of claims 1 to 8.

Images