Fruit quality grading method and system based on multi-modal model

Abstract

The invention is suitable for the technical field of agriculture, and provides a fruit quality grading method and system based on a multi-modal model, which improves the grading precision and universality by constructing a multi-modal and multi-scale feature fusion system, directly obtains accurate physicochemical index parameters such as sweetness, pH, vitamin C concentration and the like, avoids spectrum inversion errors, and improves the quality of fruits. And an appearance and internal quality combined judgment basis is formed in combination with a double-view-angle image. Meanwhile, the cross-modal interaction relation is dynamically modeled, the modal weight is adaptively distributed, and the redundancy problem of an existing fusion means is solved. In addition, global attributes and local details are synchronously captured through a pyramid structure, fruits with large size differences are compatible, and the grading performance and compatibility of the system are remarkably improved. A hierarchical model is deployed in built-in edge computing hardware of fast detection equipment, and a full-link localization closed loop of data acquisition, feature extraction, hierarchical discrimination and result display is completed. Network dependence is eliminated, off-line real-time detection is supported, and the operation and maintenance cost is greatly reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of agriculture, and in particular to a fruit quality grading method and system based on a multi-modal model. BACKGROUND

[0002] In the prior art, fruit quality grading mainly relies on a single modal detection means (such as an RGB image, a spectral image or a size parameter), which is difficult to comprehensively reflect the internal and external quality characteristics of the fruit. Although existing research attempts to realize multi-modal analysis through the fusion of multi-view images or RGB images and spectral data, the detection of internal substances (such as sweetness and PH) based on spectral inversion always has the problems of precision fluctuation and high cost: 1. Single modal detection leads to one-sided grading: The existing technology relies on a single modal such as an RGB image or a spectral image, which is difficult to comprehensively reflect the internal and external quality characteristics of the fruit. Although existing research attempts to realize multi-modal analysis through the fusion of multi-view images or RGB images and spectral data, the detection of internal substances (such as sweetness and PH) based on spectral inversion always has the problems of precision fluctuation and high cost:

[0003] 2. Low spectral inversion precision and high cost: The inversion of physical and chemical indicators based on spectra has precision errors and high hardware costs, which is difficult to popularize in small and medium-sized trading places, limiting the large-scale application of accurate grading.

[0004] 3. Multi-modal fusion method is rough: The existing fusion method only uses simple splicing or fixed weight, which cannot dynamically capture the interaction between modalities, resulting in information redundancy and reduced model generalization ability.

[0005] 4. Single feature scale, poor compatibility: The use of fixed scale features to process phenotypic data makes it difficult to capture both global attributes and local details of the fruit, and cannot adapt to fruits with large size differences.

[0006] 5. Relies on cloud computing, real-time performance is insufficient: The grading system needs cloud servers to process data, which has the problems of high delay, strong network dependence and high operation and maintenance cost, which restricts the landing of agricultural scenarios.

[0007] Therefore, it is necessary to provide a fruit quality grading method and system based on a multi-modal model to solve the above technical problems. SUMMARY

[0008] The application provides a fruit quality grading method and system based on a multi-modal model, which combines numerical values such as sweetness, PH, and vitamin C concentration directly obtained by a double-view phenotypic image and a rapid detection device, significantly improves the detection accuracy of physical and chemical parameters, and greatly reduces the detection cost through hardware integration. In addition, the application innovatively deploys the grading model in the edge computing hardware built in the rapid detection device, breaks through the limitations of traditional cloud dependence, realizes the full-link localization closed loop from data acquisition to result display, supports offline real-time detection, and provides a low-cost, high-precision, and strongly adaptive technical solution for agricultural intelligent equipment.

[0009] The application is implemented as follows: a fruit quality grading method and system based on a multi-modal model, comprising the following steps: S1, data acquisition: using an integrated rapid detection device, placing the fruit in the center of the device, collecting two types of phenotypic images of the fruit, i.e., an orthographic view and a top view, through two cameras of the same specification fixed on the device, collecting the weight of the fruit through an electronic scale integrated at the bottom end of the device, and collecting the sweetness value, pH value, and vitamin C concentration value of the fruit through sweetness, pH, and vitamin C concentration rapid detection ports integrated in the device; S2, data preprocessing: performing data cleaning on the phenotypic images, weight, sweetness value, pH value, and vitamin C concentration value collected in step S1, and re-executing step S1 for sampling if there are abnormal values or repeated data; and performing standardization processing on the cleaned data to make all types of data have the same scale; S3, multi-modal model prediction: inputting the data preprocessed in step S2 into a pre-trained multi-modal fusion prediction model to output the classification confidence of each quality grade of the fruit; the multi-modal fusion prediction model establishes the connection between the appearance phenotype, sweetness, PH, and vitamin C concentration of the fruit and the fruit quality grading, and is used for intelligently evaluating the quality grade of the fruit.

[0010] Preferably, the feature flow of the multi-modal fusion prediction model includes model input, multi-modal feature extraction, and feature fusion and classification prediction; the multi-modal fusion prediction model includes three input branches, the first branch extracts orthographic view features through a G-FPN network, the second branch extracts top view features through a G-FPN network, and the third branch extracts numerical features of the weight, sweetness value, pH value, and vitamin C concentration value through two layers of MLP; the output features of the three branches are mapped, spliced, fused by an MHSA (Multi-Head Self-Attention) module in an Encoder module, classified by a classifier composed of MLP and Softmax, and the confidence belonging to each grade is obtained to complete the quality grading.

[0011] Preferably, the network of the multi-modal feature extraction stage is divided into three branches from top to bottom: the first two branches utilize G-FPN network to extract fruit front view and top view features respectively, and the third branch utilizes MLP (Multilayer Perceptron) to extract features of weight, sweetness, PH, and vitamin C concentration vectors; the G-FPN of the two branches has the same model structure, but the model parameters thereof are independent of each other; the G-FPN network is divided into a backbone network and a multi-scale feature fusion network; The backbone network is stacked by 4 layers of Ghost residual blocks, and the specific calculation formula is as follows:

[0012] wherein, X is an input feature, Y is an output feature, represents a Ghost module, represents a Batch Normalization operation, represents a ReLU activation function, is a depth separable convolution with a stride of 2.

[0013] Further, the Ghost module calculation process is composed of three steps of operations, and the specific calculation formula of the first step is as follows:

[0014] wherein, represents a pointwise convolution; The calculation formulas of the next two steps are as follows:

[0015]

[0016] wherein, represents a concatenation operation of features in the channel dimension, , is an intermediate feature, and the output feature is .

[0017] Preferably, the feature fusion and classification prediction stage splices the three branch features after mapping, and utilizes an Encoder module for feature fusion, and finally sends into a classifier composed of MLP and Softmax for classification to obtain the confidence belonging to each grade.

[0018] Preferably, the multi-scale feature fusion network fuses and outputs the features of the last three layers of the network, and the output features of the last three layers of the backbone network are obtained after pointwise convolution to obtain multi-scale features , and , which cover large-scale features such as horizontal and vertical diameters, and capture detailed features such as local color and damage; then the three are intersected, fused and aggregated to obtain output features , and the specific calculation formula is as follows:

[0019]

[0020]

[0021]

[0022] wherein, represents a global average pooling GAP (Global Average Pooling) operation, which performs global average pooling on the spatial dimension of the feature, so that each channel outputs a feature value, and the feature is compressed from a three-dimensional matrix to a one-dimensional vector; represents a bilinear interpolation operation, which expands the deep feature space size to the same size as the shallow feature space size, so that the two can be added; as can be seen from the formula, the compressed multi-scale features are concatenated and the output of the G-FPN is obtained, corresponding to the outputs of the first and second branches and .

[0023] Preferably, in the third branch, the normalized fast detection parameter (weight, sweetness, PH, vitamin C concentration) vector is passed through a two-layer multilayer perception MLP, and then a fast detection parameter deep feature is obtained, and the specific operation formula is as follows:

[0024] wherein, represents a fully connected layer, and the output of the third branch .

[0025] Preferably, in the classification prediction stage, the output features of the three branches are concatenated in the channel dimension, and the operation formula is as follows:

[0026] wherein, the fully connected layer maps the output features of the three branches to a vector of the same length; represents a stacking operation, which stacks the mapped features in a new dimension to obtain output features ; then the features are interactively fused through the Encoder module; The specific operation of the Encoder is as follows: ①First, the input features are linearly mapped respectively:

[0027]

[0028]

[0029] wherein, represents a Layer Normalization operation; ②Get self-attention output:

[0030] wherein, , represents a Softmax function, represents self-attention; ① and ② are the calculation process of self-attention.

[0031] ③Divide Q, K, and V in ① into H parts, that is, H heads, and the output of each head is:

[0032] Then concatenate the outputs of multiple heads, and the calculation formula is as follows:

[0033]

[0034] ④The final output of the encoder is as follows:

[0035] wherein, FFN is a two-layer Feed-Forward Network; Then the fused features are interacted through an MLP, and the final classification prediction is obtained, and the specific formula is as follows:

[0036] wherein, is the model output, N is the number of fruit quality levels; the Softmax function converts the feature value into the confidence of each level.

[0037] Preferably, in the step S1, the process of collecting the vitamin C concentration value comprises: dropping the fruit juice onto the detection window of the vitamin C concentration rapid detection port, waiting for color development, rotating the equipment button to the "vitamin C content detection" option, pressing the button, and taking a photo of the vitamin C content detector with an orthographic camera, and reading the Lab color value at a fixed position (color development window) , and calculating the color value and each standard color (stored in the edge device, Color difference between the two standard colors with the smallest color difference:

[0038] Subsequently, the smallest color difference is taken 、 , and the color-concentration table is queried to obtain the vitamin C content values corresponding to the two standard colors with the smallest color difference With , the inverse of the color difference is used as the weight to evaluate the vitamin C content corresponding to the current color development: .

[0039] The present application provides a fruit quality grading system based on a multi-modal model, which is applied to the fruit quality grading method based on the multi-modal model described above, and the system comprises: A rapid detection device is used to synchronously collect multi-modal data of the fruit, including a top-view camera (1), a front-view camera (2), an electronic scale (3), a display screen (4), an operation knob (5), a vitamin C concentration rapid detector (6), a sweetness detector (7), a PH detector (8), and an LED light source (9); the top-view camera (1) and the front-view camera (2) are of the same specification and are fixed at different angles of the device, and are used to collect top-view and front-view images of the fruit; the operation knob (5) is used to switch between the functions of “image shooting”, “intrinsic substance detection”, and “quality grade prediction”; An edge computing hardware is built-in the rapid detection device module, and is used to run a data preprocessing module and a multi-modal model module, so as to realize the localization of data processing and grading discrimination, and support offline operation; The multi-modal model module is deployed in the edge computing hardware, and comprises three input branches and a feature fusion classification unit: the first branch and the second branch are G-FPN networks, which are used to extract front-view and top-view features; the third branch is a two-layer MLP, which is used to extract numerical features; the feature fusion classification unit comprises a fully connected mapping layer, an Encoder module comprising MHSA (Multi-Head Self-Attention), and an MLP classifier; A display module comprises the display screen (4) of the rapid detection device, and is used to display the classification confidence of each quality grade of the fruit.

[0040] Compared with the related art, the fruit quality grading method and system based on the multi-modal model provided by the present application have the following beneficial effects: 1. Multi-dimensional feature integrated rapid detection technology The fast detection equipment integrates internal substance detection and phenotype image collection, and can simply obtain the sweetness, PH, vitamin C concentration, weight and double-angle phenotype image of the fruit, and realize fast, non-destructive and standardized collection of multi-modal data of "internal quality + appearance".

[0041] 2. Multi-modal data fusion modeling A deep learning model fusing rational parameters and visual features is constructed to comprehensively analyze the internal quality and external morphology difference of the fruit, realize high-precision discrimination of different grades of fruit, and output classification confidence of each quality grade of the fruit.

[0042] 3. Localized real-time discrimination system of edge computing deployment The trained multi-modal model is deployed on the edge computing hardware of the fast detection equipment to realize local closed-loop processing of data collection, model inference and result display.

[0043] 4. The present application not only realizes fast and accurate grading of fruit quality grades without relying on professional fruit knowledge, without investing a large amount of manpower, and without performing tedious physicochemical detection for enterprises and consumers; but also solves the problems of randomness, subjectivity, poor consistency, high cost and low efficiency of fruit grade evaluation. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 It is a multi-modal deep network model architecture design diagram of the present application; Figure 2 It is a front view of the fast detection equipment; Figure 3 It is a fast detection equipment oblique view Figure 1 ; Figure 4 It is a fast detection equipment oblique view Figure 2 .

[0045] In the figure, the label: 1, overhead camera; 2, front camera; 3, electronic scale; 4, display screen; 5, operation knob; 6, vitamin C concentration fast detector; 7, sweetness detector; 8, PH detector; 9, LED light source. DETAILED DESCRIPTION

[0046] The present application will be further described below in combination with the drawings and embodiments.

[0047] The fruit quality grading method and system based on the multi-modal model proposed by the present application include the following steps: S1, data acquisition: using an integrated rapid detection device, placing the fruit in the center of the device, collecting two types of phenotype images of the fruit, front view and top view, through two fixed cameras of the same specification on the device, collecting the weight of the fruit through the electronic scale integrated at the bottom end of the device, collecting the sweetness value, pH value, and vitamin C concentration value of the fruit through the sweetness meter, pH meter, and vitamin C concentration rapid detection port integrated in the device; S2, data preprocessing: data cleaning is performed on the phenotype images, weight, sweetness value, pH value, and vitamin C concentration value collected in step S1, and if there are abnormal values or repeated data, step S1 is re-executed for sampling; the cleaned data is standardized to have the same scale; S3, multi-modal model prediction: input the preprocessed data in step S2 into the pre-trained multi-modal fusion prediction model to output the classification confidence of each quality grade of the fruit; the multi-modal fusion prediction model establishes the relationship between the appearance phenotype, sweetness, pH, and vitamin C concentration of the fruit and the fruit quality grading, and is used for intelligent evaluation of the quality grade of the fruit.

[0048] The feature flow of the multi-modal fusion prediction model includes model input, multi-modal feature extraction, and feature fusion and classification prediction; the multi-modal fusion prediction model includes three input branches, the first branch extracts front view features through a G-FPN network, the second branch extracts top view features through a G-FPN network, and the third branch extracts numerical features of weight, sweetness value, pH value, and vitamin C concentration through two layers of MLP; the output features of the three branches are mapped and spliced, and the MHSA (Multi-Head Self-Attention) module in the Encoder module is used for feature fusion, and then the classifier composed of MLP and Softmax is used for classification to obtain the confidence belonging to each grade and complete the quality grading.

[0049] The network in the multi-modal feature extraction stage is divided into three branches from top to bottom: the first two branches use G-FPN networks to extract fruit front view and top view features, and the third branch uses MLP (Multilayer Perceptron) to extract features of weight, sweetness, pH, and vitamin C concentration vectors; the two branches of G-FPN have the same model structure, but their model parameters are independent of each other; the model structure of G-FPN is as shown in Figure 1 b, and the G-FPN network is divided into a backbone network and a multi-scale feature fusion network; The backbone network is stacked by 4 layers of Ghost residual blocks, and the architecture of each layer of Ghost residual block is as shown in Figure 1 (c), and the specific calculation formula is as follows:

[0050] wherein, X is the input feature, Y is the output feature, represents a Ghost module, represents a Batch Normalization operation, represents a ReLU activation function, is a depthwise separable convolution with a stride of 2.

[0051] Further, an architecture diagram of the Ghost module is shown in Figure 1 (d), and the calculation process thereof is composed of three steps of operations. The specific calculation formula of the first step is as follows:

[0052] wherein, represents a point-wise convolution; The calculation formula of the next two steps is as follows:

[0053]

[0054] wherein, represents a concatenation operation of features in the channel dimension, , is an intermediate feature, and the output feature is .

[0055] The feature fusion and classification prediction stage splices the three branch features after mapping, and utilizes the MHSA (Multi-Head Self-Attention) module in the Encoder module to perform feature fusion, and finally inputs into the classifier composed of MLP and Softmax to perform classification, to obtain the confidence belonging to each level.

[0056] The multi-scale feature fusion network fuses the features of the last three layers of the network and outputs, as shown in Figure 1 (b), the output features of the last three layers of the backbone network are subjected to point-wise convolution to obtain multi-scale features , and , which cover large-scale features such as horizontal, vertical and radial, and also capture detailed features such as local color and damage; then the three are intersected, fused and aggregated to obtain the output feature , and the specific calculation formula is as follows:

[0057]

[0058]

[0059]

[0060] wherein, denotes a global average pooling GAP operation, which performs global average pooling on the spatial dimensions of the features, outputs a feature value for each channel, and compresses the features from a three-dimensional matrix to a one-dimensional vector; denotes a bilinear up-sampling operation, which expands the spatial size of the deep feature to the same size as the shallow feature so that they can be added together; as can be seen from the formula, the concatenated and obtained output of the G-FPN is and .

[0061] In the third branch, the normalized fast detection parameter (weight, sweetness, PH, and vitamin C concentration) vector is input into a two-layer multi-layer perception MLP, and then a deep feature of the fast detection parameter is obtained. The specific operation formula is as follows:

[0062] wherein, denotes a fully connected layer, and the output of the third branch is .

[0063] In the classification prediction stage, the output features of the three branches are concatenated in the channel dimension, and the operation formula is as follows:

[0064] wherein, the fully connected layer maps the output features of the three branches into vectors of the same length; denotes a stacking operation, which stacks the mapped features in a new dimension to obtain the output feature ; then the features are interactively fused through the Encoder module; The specific operation of the Encoder is as follows: ①First, the input features are linearly mapped respectively:

[0065]

[0066]

[0067] wherein, denotes a layer normalization operation; ②Obtain the self-attention output:

[0068] wherein, , denotes a Softmax function, denotes self-attention; ①, ② are the calculation processes of self-attention.

[0069] ③Divide Q, K, V in ① into H parts, i.e. H heads, and the output of each head is:

[0070] Then concatenate the outputs of multiple heads, and the calculation formula is as follows:

[0071]

[0072] ④The final output of the encoder is as follows:

[0073] wherein, FFN is a two-layer feed-forward neural network (Feed-Forward Network); Then, the fused features are interacted through an MLP to obtain the final classification prediction, and the specific formula is as follows:

[0074] wherein, is the model output, N is the number of fruit quality levels; the Softmax function converts the feature value into the relative probability of each level.

[0075] In the step S1, the collection process of the vitamin C concentration value includes: dropping the fruit juice to the detection window of the vitamin C concentration rapid detection port, waiting for color development, rotating the equipment button to the "vitamin C content detection" option, pressing the button, shooting the vitamin C content detector photo by the orthographic camera, and reading the Lab color value of the fixed position (color development window) , calculating the color difference between the color value and each standard color (stored in the edge device, is the number of standard colors)

[0076] Subsequently, the smallest color difference , is taken, and the vitamin C content values corresponding to the two standard colors with the smallest color difference are obtained by querying the color-concentration table and , using the inverse of the color difference as the weight, to evaluate the vitamin C content corresponding to the current color rendering: .

[0077] The application proposes a fruit quality grading system based on a multi-modal model, which is applied to the fruit quality grading method based on the multi-modal model described above, and the system comprises: A rapid detection device is used to synchronously collect multi-modal data of the fruit, including a top-view camera (1), a front-view camera (2), an electronic scale (3), a display screen (4), an operation knob (5), a vitamin C concentration rapid detector (6), a sweetness detector (7), a pH detector (8), and an LED light source (9); the top-view camera (1) and the front-view camera (2) are of the same specification and are fixed at different angles of the device, and are used to collect top-view and front-view images of the fruit; the operation knob (5) is used to switch between the functions of “image shooting”, “intrinsic substance detection”, and “quality grade prediction”; Edge computing hardware is built into the rapid detection device module and is used to run the data preprocessing module and the multi-modal model module, realize the localization of data processing and grading discrimination, and support offline operation; The multi-modal model module is deployed in the edge computing hardware and comprises three input branches and a feature fusion classification unit: the first branch and the second branch are G-FPN networks, which are used to extract front-view and top-view features; the third branch is a two-layer MLP, which is used to extract numerical features; the feature fusion classification unit comprises a fully connected mapping layer, an Encoder module comprising MHSA, and an MLP classifier; The display module comprises the display screen (4) of the rapid detection device and is used to display the classification confidence of each quality grade of the fruit.

[0078] Embodiment: Quality grading evaluation of Xiping green-skinned oranges Xiping County in Yuxi City, Yunnan Province is rich in “Xiping green-skinned oranges”, which are characterized by green or yellow-green skin (even when ripe), high juice content, sweet taste, and good storage and transportation performance. The evaluation of the quality of the oranges needs to pay special attention to the sugar-acid ratio, the stability of the skin color, and the fineness of the fruit flesh, and it is difficult to accurately determine the maturity and flavor using traditional methods.

[0079] Firstly, the Xiping green-skinned oranges are placed in the center of the rapid detection device, and the appearance images of the fruit are collected by the front-view and top-view cameras, the weight is synchronously obtained by the electronic scale, and the juice is dropped on the sweetness meter, the pH meter, and the vitamin C rapid detection port to complete the detection of the physicochemical indicators.

[0080] Next, the collected image data, weight, sweetness, pH, vitamin C, and other data are standardized, and if there are abnormal values, the detection process is repeated.

[0081] The quality grade prediction mode is started, image and physical and chemical characteristics are fused, the sugar-acid ratio, fruit weight and appearance grade are determined through a pre-trained model, and the final quality grade of new flat green skin orange is output, including five grades of special grade, first grade, second grade, third grade and bad fruit.

[0082] In implementation, the grading accuracy can reach 94.23%.

[0083] The technical key points of the present application are: (1) Multi-modal data synchronous acquisition mechanism An integrated rapid detection equipment is constructed, multi-dimensional data such as double-angle phenotype images, intrinsic substance parameters (sweetness, PH, vitamin C concentration), weight and the like of fruits are collected at the same time, it is ensured that the image and the physical and chemical parameters are strictly aligned in time and space, and data mismatch caused by step-by-step collection is avoided; (2) Multi-modal feature fusion model architecture A deep learning model structure of image branch + numerical branch + fusion classification head is designed to improve the discrimination accuracy and interpretability.

[0084] (3) Edge side localization real-time inference capability The trained multi-modal evaluation model is deployed on the edge computing hardware.

[0085] (4) Integrated hardware system integration The camera, weighing module, sweetness meter, edge computing unit and the like are integrated in a single device. It supports battery power supply and can be used in orchards, fields and wholesale markets without external power supply. The user interface is simple and supports "one-key detection". Ordinary operators can operate after short-term training.

[0086] Through the implementation of the present application, the following obvious effects can be brought about: Guiding fruit grading work: accurately guiding quality grading in fruit batch sampling scenes, ensuring the accuracy and consistency of fruit grade division through efficient and simplified detection methods, significantly improving sampling efficiency and reducing human error.

[0087] Providing a basis for transactions: the present application provides an objective and reliable quality basis for fruit transactions, helping buyers and sellers quickly negotiate prices based on unified standards, reducing transaction disputes, and promoting market fairness and circulation efficiency.

[0088] Providing agricultural guidance: based on the quality data of batch sampling, the present application can generate targeted agricultural recommendations, such as recommending the best harvesting time or adjusting the planting scheme, providing scientific reference for farmers to optimize production management and continuously improving fruit quality and income.

[0089] The above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the protection scope of the present application. Obviously, the described examples are only some of the embodiments of the present application, but not all the embodiments. Based on these examples, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application. Although the present application is described in detail with reference to the above examples, those of ordinary skill in the art can still combine, add or delete the features in the embodiments of the present application according to the circumstances without creative work, so as to obtain different other technical solutions which do not deviate from the concept of the present application in essence, and these technical solutions also fall within the scope of the present application.

Claims

1. A fruit quality grading method based on a multimodal model, characterized in that, Includes the following steps: S1. Data Acquisition: Using an integrated rapid testing device, the fruit is placed in the center of the device. Two cameras of the same specifications are fixed on the device to collect two types of phenotypic images of the fruit: front view and top view. The weight of the fruit is collected through an electronic scale integrated at the bottom of the device. The sweetness value, pH value, and vitamin C concentration value of the fruit are collected through the sweetness meter, pH meter, and vitamin C concentration rapid testing port integrated in the device, respectively. S2. Data preprocessing: Clean the phenotypic images, weights, sweetness values, pH values, and vitamin C concentration values ​​collected in step S1. If outliers or duplicate data are found, step S1 is repeated for sampling. Standardize the cleaned data to ensure that all types of data have the same scale. S3. Multimodal model prediction: Input the preprocessed data from step S2 into a pre-trained multimodal fusion prediction model and output the classification confidence of each quality grade of fruit. The multimodal fusion prediction model establishes the relationship between fruit appearance phenotype, sweetness, pH and vitamin C concentration and fruit quality grading, and is used to intelligently evaluate the quality grade of fruit.

2. The fruit quality grading method based on a multimodal model according to claim 1, characterized in that, The feature flow of the multimodal fusion prediction model includes model input, multimodal feature extraction, and feature fusion and classification prediction. The multimodal fusion prediction model includes three input branches: the first branch extracts front view features through a G-FPN network, the second branch extracts top view features through a G-FPN network, and the third branch extracts numerical features of weight, sweetness value, pH value, and vitamin C concentration value through a two-layer MLP. The output features of the three branches are concatenated after mapping, and feature fusion is performed using an Encoder module. Then, a classifier composed of MLP and Softmax is used for classification to obtain the confidence level of each grade, thus completing the quality grading.

3. The fruit quality grading method based on a multimodal model according to claim 2, characterized in that, The multimodal feature extraction stage network is divided into three branches from top to bottom: the first and second branches use the G-FPN network to extract the features of the front view and top view of the fruit, respectively; the third branch uses MLP to extract the features of weight, sweetness, pH, and vitamin C concentration vectors; the G-FPN of the two branches has the same model structure, but their model parameters are independent of each other; the G-FPN network is divided into a backbone network and a multi-scale feature fusion network. The backbone network is composed of 4 layers of Ghost residual blocks stacked together, and the calculation formula is as follows: ， in, X Y is the input feature, and Y is the output feature. This refers to the Ghost module. This indicates a batch normalization operation. Represents the ReLU activation function. For depthwise separable convolution with a stride of 2; Furthermore, the Ghost module's calculation process consists of three steps. The specific calculation formula for the first step is as follows: ， in, This represents pointwise convolution; The calculation formulas for the next two steps are as follows: ， ， in, This represents a concatenation operation of features along the channel dimension. , The intermediate features are used as the output features. .

4. The fruit quality grading method based on a multimodal model according to claim 2, characterized in that, The feature fusion and classification prediction stage concatenates the three branch features after mapping, and uses the Encoder module to perform feature fusion. Finally, it is fed into a classifier composed of MLP and Softmax for classification to obtain the confidence level of each level.

5. The fruit quality grading method based on a multimodal model according to claim 4, characterized in that, The multi-scale feature fusion network fuses and outputs the features from the last three layers of the network. The output features from the last three layers of the backbone network are then convolved pointwise to obtain the multi-scale features. , and These three elements encompass large-scale features in both horizontal and vertical dimensions, and also capture local color and damage details. Next, the features are fused and aggregated to obtain the output features. The calculation formula is as follows: ， ， ， ， in, This represents the Global Average Pooling (GAP) operation, which performs global average pooling on the spatial dimension of the features, so that each channel outputs a feature value, compressing the features from a three-dimensional matrix to a one-dimensional vector. This represents a bilinear up-interpolation operation that expands the deep feature space to the same size as the shallow feature space so that they can be added together. The compressed multi-scale features are concatenated to obtain the output of the G-FPN, corresponding to the outputs of the first and second branches. and .

6. The fruit quality grading method based on a multimodal model according to claim 1, characterized in that, In the third branch, the standardized fast detection parameter vector is passed through a two-layer multilayer perceptron (MLP) to obtain the deep features of the fast detection parameters. The specific calculation formula is as follows: ， in, This represents the output of the third branch of a fully connected layer. .

7. The fruit quality grading method based on a multimodal model according to claim 1, characterized in that, In the classification prediction stage, the output features of the three branches are concatenated along the channel dimension, and the calculation formula is as follows: ， The fully connected layer maps the output features of the three branches to vectors of the same length. This represents a stack operation, which stacks the mapped features in a new dimension to obtain the output features. Next, the features are interactively fused using the Encoder module. The specific operations of the Encoder are as follows: ① First, perform linear mapping on the input features respectively: ， ， ， in, Presentation layer normalization operation; ② Obtain self-attention output: ， in, , This represents the Softmax function. Indicates self-attention; ① and ② are the calculation process of self-attention; ③ Divide Q, K, and V from ① into H parts, i.e., H heads. Then the output of each head is: ， Next, the outputs of multiple heads are concatenated, and the calculation formula is as follows: ， ， ④The final output of the Encoder is as follows: ， FFN is a two-layer feedforward neural network; Next, the fused features are interacted using an MLP to obtain the final hierarchical classification prediction. The specific formula is as follows: ， in, The model output is N, which represents the number of fruit quality levels. The Softmax function converts the feature values ​​into confidence scores for each level.

8. The fruit quality grading method based on a multimodal model according to claim 1, characterized in that, In step S1, the process of collecting the vitamin C concentration value includes: dropping fruit juice onto the detection window of the vitamin C concentration rapid detection port, waiting for color development, rotating the device button to the "Vitamin C Content Detection" option, pressing the button, and having the orthophoto camera take a picture of the vitamin C content detector and read the Lab color value at a fixed position (color development window). Calculate color values ​​and each standard color. (Stored on edge devices,) Color difference between the standard color quantity and the standard color quantity: ， Then, take the smallest color difference. , And by consulting the color-concentration table, we can obtain the vitamin C content values ​​corresponding to the two standard colors with the smallest color difference. and Using the reciprocal of the color difference as a weight, the vitamin C content corresponding to the current color is evaluated: 。 9. A fruit quality grading system based on a multimodal model, characterized in that, This system is applied to the fruit quality grading method based on a multimodal model as described in any one of claims 1-8, the system comprising: The rapid testing equipment is used to simultaneously collect multimodal data of fruits, including a top-view camera (1), a front-view camera (2), an electronic scale (3), a display screen (4), an operation knob (5), a vitamin C concentration rapid tester (6), a sweetness tester (7), a pH tester (8), and an LED light source (9); the top-view camera (1) and the front-view camera (2) are of the same specification and fixed at different angles of the equipment, used to collect top-view and front-view views of fruits; the operation knob (5) is used to switch between "image capture", "intrinsic substance detection", and "quality grade prediction" functions; Edge computing hardware, built into the rapid detection device module, is used to run the data preprocessing module and the multimodal model module, realizing localized data processing and classification, and supporting offline operation; The multimodal model module, deployed in edge computing hardware, includes three input branches and a feature fusion classification unit: the first and second branches are G-FPN networks used to extract front view and top view features, and the third branch is a two-layer MLP used to extract numerical features. The feature fusion classification unit includes a fully connected mapping layer, an encoder module with MHSA (Multi-Head Self-Attention), and an MLP classifier. The display module includes a display screen (4) of the rapid testing equipment, used to display the classification confidence level of each quality grade of fruit.

Images