Seed quality evaluation system and device based on multi-modal data fusion

Abstract

The invention discloses a seed quality evaluation system and device based on multi-modal data fusion and a control method, and relates to the technical field of agricultural engineering. The system comprises a data acquisition module, a data processing module, a quality evaluation module and a control module. The data acquisition module acquires multi-modal data; the data processing module performs preprocessing, feature extraction and fusion on the multi-modal data to generate a comprehensive feature vector; the quality evaluation module performs seed quality evaluation based on the comprehensive feature vector by using a machine learning model; the control module controls all the modules to operate and outputs results. The device comprises a closed box body, a temperature and humidity control unit, an electronic scale, a hyperspectral imaging unit, a visible light imaging unit, a mechanical arm and a main control unit. The method realizes rapid, accurate, lossless and automatic evaluation of seed quality, overcomes the defects of low efficiency, strong subjectivity, easy damage of seeds and the like in a traditional method, and improves the accuracy and efficiency of seed quality evaluation.

Description

Technical Field

[0001] The present invention relates to the field of agricultural engineering technology, and in particular to a seed quality assessment system, device and control method based on multimodal data fusion. Background Art

[0002] Seed quality is a key factor in agricultural production and seed trade. High-quality seeds can increase crop yields, improve crop quality, and enhance crop stress resistance, thereby ensuring food security and sustainable agricultural development. Traditional seed quality testing methods mainly include physical testing, chemical testing, and biological testing.

[0003] Physical detection methods, such as appearance inspection and thousand-grain weight measurement, are easy to operate, but they are highly subjective and have low accuracy, making it difficult to fully reflect seed quality. Chemical detection methods, such as moisture measurement and component analysis, have accurate results, but they are complex to operate, time-consuming, and destructive, and can damage seeds. Biological detection methods, such as germination tests and accelerated aging tests, can reflect the physiological activity of seeds, but they take longer, have high requirements for environmental conditions, and the results are easily affected by environmental factors.

[0004] In recent years, with the development of spectral imaging technology, non-destructive seed quality detection methods based on near-infrared spectroscopy and hyperspectral imaging have gradually attracted attention. Although near-infrared spectroscopy technology can quickly determine the content of some components in seeds, its spectral resolution is low and it is difficult to capture more subtle quality differences inside the seeds. Although hyperspectral imaging technology can simultaneously obtain spectral information and image information of seeds and provide richer quality information, the data volume is large, the processing is complex, the equipment cost is high, and the model generalization ability needs to be improved. Existing seed quality detection systems usually only focus on data of a single modality, which limits the detection accuracy and reliability.

[0005] Therefore, it is necessary to improve the existing seed quality assessment technology to overcome the shortcomings of the existing technology. Summary of the invention

[0006] In order to overcome the problems existing in the related art, one of the purposes of the present invention is to provide a seed quality assessment system based on multimodal data fusion. The system integrates multimodal data acquisition modules such as hyperspectral imaging, visible light imaging and weight sensing, and combines data preprocessing, feature extraction, feature fusion and machine learning algorithms to build an automated, intelligent and integrated seed quality assessment system to achieve rapid, accurate, non-destructive and automated assessment of seed quality.

[0007] A seed quality assessment system based on multimodal data fusion, comprising: A data acquisition module, used for acquiring multimodal data of seeds, wherein the multimodal data includes a spectral image of the seeds, a visible light image of the seeds, and weight data of the seeds; A data processing module, used for preprocessing, feature extraction and feature fusion of the multimodal data to generate a comprehensive feature vector; A quality assessment module, using a machine learning model to perform seed quality assessment based on the comprehensive feature vector to obtain a seed quality assessment result; A control module, used to control the operation of the data acquisition module, the data processing module and the quality assessment module, and output the seed quality assessment result; The preprocessing includes performing black and white correction, filtering and scattering correction on the spectral image, and performing target detection annotation and data enhancement on the visible light image.

[0008] By integrating data acquisition module, data processing module, quality assessment module and control module, it is possible to comprehensively utilize the spectral image, visible light image and weight data of seeds and analyze them using machine learning models, thereby achieving a comprehensive, accurate and automated assessment of seed quality. Compared with traditional single-modal data or manual assessment methods, the system can more objectively and efficiently reflect the quality characteristics of seeds, providing a reliable basis for seed production, processing, storage, transportation and use.

[0009] Furthermore, the data processing module includes: A hyperspectral data processing unit, used for performing black-and-white correction, Savitzky-Golay filtering and Multiplicative Scatter Correction on the spectral image, and outputting hyperspectral features; A visible light image processing unit, used for labeling and data enhancement of the visible light image for target detection, and outputting visible light image features, wherein the data enhancement adopts a conditional generative adversarial network cGAN; The feature fusion unit is used to fuse the hyperspectral feature, the visible light image feature and the weight data to generate the comprehensive feature vector.

[0010] Through the collaborative work of the hyperspectral data processing unit, the visible light image processing unit and the feature fusion unit, the collected multimodal data can be effectively preprocessed, feature extracted and fused to generate high-quality comprehensive feature vectors, providing a reliable data basis for subsequent quality assessment. The use of Savitzky-Golay filtering and MultiplicativeScatter Correction scattering correction can effectively remove noise and scattering interference in spectral images, and the use of conditional generative adversarial networks cGAN for data enhancement can improve the robustness and generalization ability of visible light image features.

[0011] Furthermore, the data processing module can adopt different preprocessing, feature extraction and feature fusion algorithms. The preprocessing can include denoising, smoothing, normalization, etc., the feature extraction can include principal component analysis (PCA), linear discriminant analysis (LDA), etc., and the feature fusion can include weighted averaging, splicing, feature-level fusion, etc.

[0012] Furthermore, the hyperspectral data processing unit can adopt different black and white correction methods, such as correction methods based on standard reference plates or correction methods based on the image itself. The window size and polynomial order of the Savitzky-Golay filter can be adjusted according to the noise level of the actual spectral data; the MultiplicativeScatter Correction scattering correction can adopt different reference spectra, such as the average spectrum of all samples or the average spectrum of samples of a specific category. The target detection annotation of the visible light image processing unit can adopt different annotation tools and annotation formats, such as LabelImg, Labelme, etc., and can adopt annotation formats such as PASCAL VOC and COCO. The generator and discriminator of the conditional generative adversarial network cGAN can adopt different network structures, such as convolutional neural network (CNN), recurrent neural network (RNN), etc.

[0013] Furthermore, the quality assessment module includes: A moisture content prediction unit, using a support vector machine model to predict the moisture content of seeds based on the hyperspectral features; A seed classification unit, using a YOLOv8 model, classifies seeds based on the visible light image features, and the seed categories include full grains, empty shell grains, barren grains, impurities, and bad grains; The quality grade evaluation unit uses a random forest model to evaluate the quality grade of the seeds based on the comprehensive feature vector.

[0014] Through the collaborative work of the moisture content prediction unit, seed classification unit, and quality grade assessment unit, different machine learning models can be used to evaluate multiple key quality indicators of seeds. Among them, the support vector machine model is good at processing nonlinear relationships and is suitable for moisture content prediction; the YOLOv8 model has efficient target detection capabilities and is suitable for seed classification; the random forest model has strong noise resistance and generalization capabilities and is suitable for quality grade assessment. Through the combination of these models, a comprehensive and accurate assessment of seed quality can be achieved.

[0015] Furthermore, the quality assessment module can adopt different machine learning models. In addition to the mentioned SVM, YOLOv8, and random forest, other models can also be adopted, such as neural networks, decision trees, support vector regression (SVR), etc.

[0016] Furthermore, the support vector machine model can use different kernel functions. In addition to the RBF kernel function, linear kernel functions, polynomial kernel functions, etc. can also be used. The network structure and hyperparameters of the YOLOv8 model can be adjusted according to the actual application scenario. Different sizes of input images, different anchor box sizes, different learning rates, etc. can be used. The number of decision trees and the maximum depth of the random forest model can be adjusted according to the size and complexity of the data set. The quality level assessment unit can classify the seed quality into different levels, for example, it can be divided into multiple levels such as excellent, good, medium, and poor, or it can be divided into two levels: qualified and unqualified.

[0017] Furthermore, the data acquisition module includes: A hyperspectral imaging unit, used for collecting spectral images of seeds, wherein the spectral range collected by the hyperspectral imaging unit is 953.5nm to 1720.23nm; A visible light imaging unit, used for collecting visible light images of seeds; Weight sensor, used to collect seed weight data.

[0018] By collecting seed spectral images in the 953.5nm to 1720.23nm band through the hyperspectral imaging unit, rich spectral information of seeds in the near-infrared band can be obtained, which is used to identify the internal quality of seeds; by collecting visible light images of seeds through the visible light imaging unit, information such as the appearance, morphology, and color of seeds can be obtained, which is used to identify external defects of seeds; by collecting seed weight data through the weight sensor, information such as the density and fullness of seeds can be obtained, which is used to evaluate the overall quality of seeds.

[0019] Furthermore, the data acquisition module can use different types of sensor combinations. In addition to spectral images and visible light images, X-ray images, thermal imaging images, etc. can be added to further enrich the types of multimodal data. Spectral images can use hyperspectral cameras with different bands, visible light images can use color cameras or black and white cameras, and weight data can use high-precision electronic scales or weighing sensors.

[0020] Furthermore, the hyperspectral imaging unit can use different types of spectrometers and cameras, for example, a line scan hyperspectral camera, an array hyperspectral camera, a push-broom hyperspectral camera, etc. The visible light imaging unit can use different types of cameras and light sources, for example, a color camera, a black and white camera, an industrial camera, etc., and can use LED light sources, halogen light sources, xenon light sources, etc. The weight sensor can use different types of sensors and ranges, for example, a strain gauge weighing sensor, a piezoelectric weighing sensor, a capacitive weighing sensor, etc., and the appropriate range can be selected according to the weight range of the seeds to be measured.

[0021] Furthermore, the YOLOv8 model adopts GIoU Loss as the loss function, uses NMS for post-processing, uses L1 regularization for parameter pruning, and uses knowledge distillation for model lightweighting.

[0022] The support vector machine model adopts the RBF kernel function and uses Bayesian optimization for parameter tuning.

[0023] The random forest model outputs the final classification result through a voting mechanism.

[0024] The YOLOv8 model uses GIoU Loss as the loss function, which can improve the accuracy and stability of target detection; NMS is used for post-processing to remove duplicate detection frames and improve the accuracy of detection results; L1 regularization is used for parameter pruning to reduce the number of model parameters and improve the model running speed; knowledge distillation is used for model lightweighting, which can further reduce the model size while ensuring the accuracy of the model, making it easier to deploy on resource-constrained devices. The support vector machine model uses the RBF kernel function to process nonlinear data; Bayesian optimization is used for parameter tuning to automatically find the optimal hyperparameter combination and improve model performance. The random forest model outputs the final classification result through a voting mechanism, which can improve the robustness and accuracy of the classification.

[0025] The second object of the present invention is to provide a seed quality assessment device based on multimodal fusion, which is used to implement the seed quality assessment system based on multimodal data fusion as mentioned above, comprising: Sealed box; A temperature and humidity control unit, used to control the temperature and humidity in the sealed box; An electronic scale, used for placing the seeds to be tested and measuring the weight of the seeds to be tested; A hyperspectral imaging unit, comprising a hyperspectral camera and a halogen lamp, wherein the hyperspectral camera is used to collect spectral images of the seeds to be tested; A visible light imaging unit, comprising a visible light camera and a bottom light source, wherein the visible light camera is used to collect visible light images of the seeds to be tested, and the bottom light source is placed on the electronic scale; A mechanical arm, used for grabbing and placing a container for holding the seeds to be tested; The main control unit includes the data processing module, the quality assessment module and the control module, and is communicatively connected with the temperature and humidity control unit, the hyperspectral imaging unit, the visible light imaging unit, the electronic scale and the robotic arm.

[0026] The temperature and humidity control unit, the hyperspectral imaging unit, the visible light imaging unit, the electronic scale and the mechanical arm are all arranged in the closed box.

[0027] A relatively stable and controllable environment is provided by a closed box, reducing the impact of external light and temperature and humidity changes on seed quality assessment; the temperature and humidity control unit can control the temperature and humidity in the box within a preset range to ensure the accuracy and repeatability of the assessment results; the electronic scale is used to measure the weight of the seeds to be tested and provide weight data to the main control unit, and the hyperspectral imaging unit and the visible light imaging unit respectively collect spectral images and visible light images of the seeds to be tested, providing multimodal data to the main control unit; the robotic arm can automatically grab and place containers containing seeds to be tested to achieve automated operation; the main control unit integrates a data processing module, a quality assessment module and a control module, which can process and analyze the collected data and control the operation of the entire device to achieve automated seed quality assessment.

[0028] Furthermore, the sealed box can be made of different materials and structures, for example, metal, plastic or glass, and can be designed into different shapes and sizes, for example, square, round, etc. The temperature and humidity control unit can use different sensors and actuators, for example, temperature and humidity sensors, heaters, refrigerators, humidifiers, dehumidifiers, etc. The electronic scale can use sensors with different precision and range. The hyperspectral imaging unit and the visible light imaging unit can use different types of cameras and light sources. The robotic arm can use different types of robotic arms, for example, industrial robots, collaborative robots, multi-joint robotic arms, SCARA robotic arms, etc. The main control unit can use different hardware platforms and software architectures, for example, embedded systems, single-chip microcomputers, PLCs, computers, cloud computer systems, etc. can be used as hardware platforms. A lighting device can be set in the box to provide uniform lighting. A camera can be set in the box to monitor the experimental process.

[0029] Furthermore, it also includes a fixed frame, which is arranged in the closed box; The fixed frame includes a first bracket arranged vertically, and the first bracket is connected to the closed box; The first bracket is also provided with a second bracket arranged horizontally; The hyperspectral imaging unit and the visible light camera are arranged on the second bracket.

[0030] The fixed frame is set in the closed box, providing stable support for the hyperspectral imaging unit and the visible light imaging unit, ensuring the imaging quality. The vertically arranged first bracket connects the closed box and the horizontally arranged second bracket, on which the hyperspectral imaging unit and the visible light camera are arranged. This structural design allows the imaging unit to be easily adjusted in height and angle to adapt to seed samples of different sizes and shapes.

[0031] Furthermore, the fixed frame can be made of different materials and structures, for example, metal materials, plastic materials, carbon fiber materials, etc., and can be made of frame structures, truss structures, etc. The first bracket and the second bracket can be connected in different ways, for example, bolt connection, welding, snap connection, etc. The second bracket can be designed to have an adjustable height and angle structure, for example, slide rails, gears, motors, etc. can be used to achieve adjustment. The installation position of the hyperspectral imaging unit and the visible light camera on the second bracket can be adjusted according to actual needs.

[0032] Furthermore, a turntable is provided on the second bracket, the hyperspectral camera and the visible light camera are fixed on the turntable, and the halogen lamp is fixed on a side of the hyperspectral camera.

[0033] The second bracket is provided with a turntable, on which the hyperspectral camera and visible light camera are fixed, and the halogen lamp is fixed on the side of the hyperspectral camera. The turntable can drive the hyperspectral camera and visible light camera to rotate, so as to realize imaging of seed samples at different angles, thereby obtaining more comprehensive information. The halogen lamp is fixed on the side of the hyperspectral camera, and as the light source of hyperspectral imaging, it can provide stable and uniform lighting to ensure the quality of spectral images.

[0034] Furthermore, the turntable can be driven in different ways, for example, motor driven, manual driven, etc. The hyperspectral camera and visible light camera can be fixed on the turntable by bolt connection, snap connection, etc. The number and power of the halogen lamps can be adjusted according to actual needs. Multiple installation positions can be set on the turntable to accommodate different types of imaging units. Filters can be added in front of the halogen lamps to adjust the spectral range or remove light of a specific wavelength.

[0035] A third object of the present invention is to provide a control method for a seed quality assessment device, which is used to control the seed quality assessment device based on multimodal fusion as described above, comprising the following steps: Controlling the temperature and humidity control unit to adjust the temperature and humidity in the sealed box to preset values; Controlling the mechanical arm to place the container containing the seeds to be tested on the electronic scale; Controlling the electronic scale to measure the weight data of the seeds to be tested, and sending the weight data to the main control unit; Controlling the hyperspectral imaging unit to collect the spectral image of the seed to be tested, and sending the spectral image to the main control unit; Controlling the visible light imaging unit to collect the visible light image of the seed to be tested, and sending the visible light image to the main control unit; The main control unit receives the weight data, the spectral image and the visible light image, and processes and analyzes the weight data, the spectral image and the visible light image to obtain a seed quality assessment result; The main control unit outputs the seed quality assessment result.

[0036] By controlling the temperature and humidity control unit to adjust the temperature and humidity in the closed box to the preset values, a stable environmental condition is provided for seed quality assessment; by controlling the robotic arm to automatically grab and place the container containing the seeds to be tested, automated operation is realized and the assessment efficiency is improved; by controlling the electronic scale, hyperspectral imaging unit and visible light imaging unit to respectively collect the weight data, spectral image and visible light image of the seeds, and sending the data to the main control unit, the automatic collection of multimodal data is realized; the main control unit processes and analyzes the received data, obtains the seed quality assessment results and outputs them, realizing the automation and intelligence of seed quality assessment.

[0037] Furthermore, the weight data, spectral image and visible light image are processed and analyzed to obtain a seed quality assessment result, comprising the following steps: Performing black-and-white correction, Savitzky-Golay filtering and Multiplicative ScatterCorrection scattering correction on the spectral image to obtain a hyperspectral feature; Performing data enhancement on the visible light image, wherein the data enhancement adopts a conditional generative adversarial network cGAN, and then performing target detection annotation to obtain visible light image features; Fusing the hyperspectral features, visible light image features and weight data to generate a comprehensive feature vector; Using a support vector machine model, predicting the moisture content of seeds based on the hyperspectral features; Using the YOLOv8 model, the seeds are classified based on the visible light image features, and the seed categories include full grains, empty shell grains, barren grains, impurities and bad grains; The quality grade of the seeds is evaluated based on the comprehensive feature vector using a random forest model.

[0038] The main control unit performs black and white correction, Savitzky-Golay filtering and Multiplicative Scatter Correction on the spectral image, which can effectively remove noise and interference in the spectral data and improve the quality of the spectral data; the conditional generative adversarial network cGAN is used to enhance the data of the visible light image, and then the target detection is annotated, which can expand the data set and improve the annotation efficiency and accuracy; the hyperspectral features, visible light image features and weight data are fused to generate a comprehensive feature vector, which can complement the advantages of different modal data and provide more comprehensive and effective information for subsequent quality assessment; the support vector machine model, YOLOv8 model and random forest model are used to predict moisture content, classify seeds and evaluate quality grades respectively, which can achieve a comprehensive evaluation of seed quality in many aspects and at multiple levels.

[0039] The beneficial effects of the present invention are: The present invention provides a seed quality assessment system based on multimodal data fusion, which includes four modules: a data acquisition module, a data processing module, a quality assessment module, and a control module, and constructs a complete seed quality assessment solution. The system obtains multiple modal information such as spectral images, visible light images, and weight data of seeds through the data acquisition module, and realizes a comprehensive characterization of the internal quality and external characteristics of seeds; the data processing module preprocesses, extracts features, and fuses features of multimodal data, effectively eliminates noise and redundant information in the data, and integrates the advantages of different modal data to generate a comprehensive feature vector that can comprehensively and accurately reflect the quality of seeds; the quality assessment module uses a machine learning model to evaluate seed quality based on a comprehensive feature vector, overcomes the defects of traditional methods such as strong subjectivity, low efficiency, and easy damage to seeds, and realizes the intelligence and automation of seed quality assessment; the control module coordinates and controls the operation of each module, and outputs the evaluation results, ensuring the overall coordination and efficient operation of the system. The system can not only improve the accuracy and efficiency of seed quality assessment, but also has the advantages of non-destructive testing, high degree of automation and strong scalability. It can meet the needs of modern agricultural production for fast, accurate and non-destructive testing of seed quality, and provide important technical support for seed production, processing, storage, transportation, sales and use. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is an architecture diagram of the cGAN provided in Example 1 of the present application; Figure 2 is an external view of a seed quality assessment device based on multimodal data fusion provided in the present application; Figure 3is an internal view of a seed quality assessment device based on multimodal data fusion provided in the present application; Figure 4 It is a structural schematic diagram of the temperature and humidity adjustment system provided in this application; Figure 5 It is a schematic diagram of the inner wall structure of the closed box provided in this application; Figure 6 It is a flowchart of the multimodal data fusion processing provided in this application.

[0041] Reference numerals: 1. Sealed box; 2. High-definition display screen; 3. Servo motor; 4. Visible light camera; 5. Halogen lamp; 6. Hyperspectral camera; 7. Robotic arm; 8. Transparent container; 9. LED bottom light source; 10. Electronic scale; 11. Background board; 12. Calibration board; 13. 360° camera; 14. Temperature sensor; 15. Humidity sensor; 16. Temperature and humidity control system; 17. Temperature regulator; 18. Humidity regulator. DETAILED DESCRIPTION

[0042] The preferred embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.

[0043] Example 1 like Figures 1 to 6 As shown, this embodiment provides a seed quality assessment system, device and control method based on multimodal data fusion, which are used to quickly, accurately and non-destructively assess the quality of seeds.

[0044] The seed quality assessment system of this embodiment includes a data acquisition module, a data processing module, a quality assessment module and a control module.

[0045] The data acquisition module includes a hyperspectral imaging unit, a visible light imaging unit and a weight sensor.

[0046] The hyperspectral imaging unit includes a hyperspectral camera 6 (spectral range 953.5nm to 1720.23nm) and two halogen lamps 5. The halogen lamps 5 are symmetrically arranged on both sides of the hyperspectral camera 6 and parallel to the hyperspectral camera 6 to provide uniform illumination.

[0047] The visible light imaging unit includes a color CCD camera and an LED bottom light source 9 .

[0048] The weight sensor is a high-precision electronic scale 10.

[0049] The data processing module includes a hyperspectral data processing unit, a visible light image processing unit and a feature fusion unit. The functions of these units are implemented through software in an embedded system or an industrial computer.

[0050] The hyperspectral data processing unit implements the following processes: 1. Black and white correction: Before collecting spectral images each time, collect a completely black reference image ( , light source off) and a completely white reference image ( , using a standard white board), and then according to formula (1) the original spectral image ( ) is corrected to obtain the corrected spectral image ( ): .

[0051] 2. ROI extraction and average spectrum calculation: Using the OpenCV library, the seed area is extracted from the corrected spectral image as the region of interest (ROI) through image processing algorithms (such as threshold segmentation and edge detection), and the average spectrum of all pixels in the ROI is calculated.

[0052] 3. Characteristic wavelength selection: 970nm, 1200nm, and 1450nm were selected as characteristic wavelengths.

[0053] 4. Preprocessing: The average spectral data were processed by Savitzky-Golay (SG) filtering and Multiplicative ScatterCorrection (MSC).

[0054] SG filtering: Use an X-order polynomial (e.g., 3rd order) and a Y-point window (e.g., 11 points) for smoothing filtering, and smooth the spectral data according to formula (2): ; in, is the smoothed spectrum value, is the SG filter coefficient, is half the size of the window, is the wavelength The spectral value at It is wavelength, is the wavelength index.

[0055] MSC: The average spectrum of all samples is selected as the reference spectrum, and the spectrum of each sample is corrected according to formula (3): ; in, is the corrected spectrum, is the original spectrum, and are the mean and standard deviation of the sample spectra, and are the mean and standard deviation of the reference spectrum, respectively.

[0056] The visible light image processing unit implements the following processes: 1. Image annotation: The LabelImg tool is used to annotate the collected visible light images. The annotated categories include full kernels, empty shell kernels, barren kernels, impurities (such as stones), and bad kernels.

[0057] 2. Data enhancement: The conditional generative adversarial network (cGAN) is used to perform data augmentation on the annotated images to expand the dataset. The generator and discriminator network structures of cGAN are as follows: The generator consists of a fully connected layer and four convolutional layers. The first four layers use the ReLU activation function and the output layer uses the Tanh activation function.

[0058] The discriminator consists of four convolutional layers and a fully connected layer. The convolutional layer uses the Leaky ReLU activation function, and the fully connected layer uses the Sigmoid activation function.

[0059] The input of the generator is random noise z and conditional label c (one-hot encoding, dimension 3, representing empty shell kernels, chaff kernels, and bad kernels respectively), and the output is the generated image. The input of the discriminator is the image and conditional label c, and the output is the probability that the image is a real sample.

[0060] The feature fusion unit implements the following process: The hyperspectral features (spectral values ​​of the average spectrum at 970nm, 1200nm, and 1450nm after preprocessing), visible light image features (the proportion of the number of seeds in each category to the total number of detections), and weight data (thousand-grain weight) were concatenated to form a comprehensive feature vector.

[0061] The quality assessment module includes a moisture content prediction unit, a seed classification unit, and a quality grade assessment unit. The functions of these units are implemented through software in an embedded system (or an industrial computer, modified according to actual conditions).

[0062] 1. Moisture content prediction unit, which predicts the moisture content of seeds based on the support vector machine (SVM) regression model.

[0063] The model training uses the hyperspectral features of seed samples with known moisture content as input and moisture content as output to train the SVM regression model. The radial basis function (RBF) is used as the kernel function, and the kernel function parameters ( ) is optimized by Bayesian optimization method. The SVM regression model formula is as follows: ; ; in, is the predicted moisture content, is the weight vector, is the mapping function, is the input feature vector (hyperspectral feature), is the bias term, is the RBF kernel function, is the sample vector, is the sample vector and The Euclidean distance between is the parameter of the kernel function (controlling the width of the function).

[0064] By inputting the hyperspectral features of the seeds to be tested into the trained SVM model, the predicted moisture content can be obtained.

[0065] 2. Seed classification unit, classifies seeds based on the YOLOv8 model.

[0066] Model training uses labeled visible light images as training data to train the YOLOv8 model. The network structure of the YOLOv8 model consists of three parts: Backbone, Neck, and Head.

[0067] Backbone adopts a series of convolutional and deconvolutional layers and uses residual connections and bottleneck structures for feature extraction.

[0068] Neck uses the SPPF module for multi-scale pooling, the PAA module for anchor box assignment, and the PAN module for feature fusion.

[0069] The head includes a detection head and a classification head, which are used to predict bounding boxes and category probabilities, respectively.

[0070] Insert the CBAM (Convolutional Block Attention Module) attention mechanism module between the feature fusion layers. Use the Adam optimizer for training and use GIoU Loss as the loss function. The formula is as follows: ; in, is the prediction box, It is a real frame. is included and The minimum closure area of ​​, is the intersection-and-union ratio, It is Generalized Intersection over Union, a loss function used for object detection.

[0071] After training, non-maximum suppression (NMS) is used for post-processing.

[0072] Model optimization uses L1 regularization to prune model parameters and uses knowledge distillation to lightweight the model. The knowledge distillation process is as follows: Prepare a pre-trained teacher model (T) and a lighter-weight student model (S).

[0073] Feed the same batch of training data into both the teacher model and the student model.

[0074] Calculate the total loss function of the student model: ; in, is the total loss, It's a mission loss. is the classification distillation loss, is the feature distillation loss, and is the weight coefficient.

[0075] ; in, It's a mission loss. is the classification loss, are the bounding box loss and the confidence loss.

[0076] The classification distillation loss is calculated using KL divergence: ; in, is the distillation temperature (for example, ), is the knowledge distillation loss (KL divergence is used here), is the probability distribution output by the teacher model elements, The probability distribution output by the student model (StudentModel) elements.

[0077] The feature distillation loss is calculated using L2 loss: ; in, is the total number of feature map elements, is the feature map loss, is the student model feature, is the teacher model feature.

[0078] The parameters of the student model are optimized through the gradient back-propagation algorithm.

[0079] Seed classification: Input the visible light image of the seeds to be tested into the trained YOLOv8 model to obtain the seed category (full grains, empty shelled grains, barren grains, impurities, and bad grains).

[0080] 3. Quality grade assessment unit, which assesses the quality grade of seeds based on the random forest model.

[0081] The model training uses the comprehensive feature vector as input and the seed quality level (high, medium, low) as output to train the random forest model. The random forest model consists of multiple decision trees, each of which is generated by bootstrap sampling.

[0082] The comprehensive feature vector of the seed to be tested is input into the trained random forest model. Each decision tree will output a prediction result. The final quality level is determined by the voting mechanism. The final classification result is obtained by formula (11): ; in, is the prediction result (quality level), Indicates majority vote, is the prediction result of each decision tree, is the input feature vector.

[0083] 4. Control module: uses embedded system (or industrial computer) as the control center, and realizes control and coordination of each module through software programming.

[0084] The seed quality assessment device of this embodiment includes the following components: 1. The sealed box 1 provides a relatively isolated and stable environment for the entire system, reducing the influence of external light, temperature, humidity and other factors on the measurement results. The material of the sealed box 1 can be black acrylic plate or other opaque materials with certain heat insulation properties, such as aluminum alloy, spray-coated steel plate, etc.

[0085] The box body adopts a frame structure or one-piece molding, and the size is determined by the size and layout of the internal components. A door that can be opened and closed is set on the front of the box body to facilitate the placement of samples and maintenance of equipment. An observation window is set on the side or top of the box body to facilitate observation of the internal situation. The bottom of the box body is provided with height-adjustable feet for leveling the device. The various panels are connected by bolts (or other connection methods, such as gluing, snaps, etc.).

[0086] 2. A temperature and humidity control unit is used to obtain and adjust the temperature and humidity in the sealed box 1 to keep it within a preset range, thereby providing a stable environmental condition for seed quality assessment. It includes a temperature sensor 14, a humidity sensor 15, a heater, a refrigerator, a humidifier, and a dehumidifier. The temperature sensor 14 and the humidity sensor 15 are used to monitor the temperature and humidity in the box in real time.

[0087] The heater may be a PTC ceramic heater such as a resistance wire heater, a far infrared heater, etc., which is used to increase the temperature inside the box.

[0088] The refrigerator may adopt a semiconductor cooling sheet, or other types of refrigerators, such as a compressor refrigerator, etc., to reduce the temperature inside the box.

[0089] The humidifier may be an ultrasonic humidifier, or other types of humidifiers, such as a steam humidifier, etc., to increase the humidity in the box.

[0090] The dehumidifier may be a semiconductor refrigeration dehumidifier, or other types of dehumidifiers, such as a rotary dehumidifier, etc., to reduce the humidity in the box.

[0091] The temperature sensor 14 , the humidity sensor 15 , the heater, the refrigerator, the humidifier and the dehumidifier are all electrically connected to the main control unit, and the main control unit controls the operation of the heater, the refrigerator, the humidifier and the dehumidifier according to the feedback signals of the temperature sensor 14 and the humidity sensor 15 .

[0092] 3. An electronic scale 10 is used to place and weigh the seeds to be tested, and transmit the weight data to the main control unit. The range, accuracy and size of the electronic scale 10 need to be selected according to actual needs, and it is necessary to ensure that there is a sufficiently stable weighing plane, and sufficient measurement range and accuracy. It is connected to the main control unit through a serial port or other communication interface, such as USB, 485, etc.

[0093] 4. A fixed frame is used to support the hyperspectral imaging unit and the visible light imaging unit. It can be made of aluminum alloy, steel or other materials with sufficient strength.

[0094] The fixed frame includes two first brackets arranged vertically to provide height support.

[0095] The fixed frame also includes a horizontally arranged second bracket, which is vertically connected to the two first brackets and is arranged between the two first brackets.

[0096] The fixed frame also includes a rotatable turntable arranged on the second bracket, which is used to drive the hyperspectral camera 6 and the visible light camera 4 to rotate synchronously, and the turntable is driven by a servo motor 3.

[0097] The first bracket is fixed to the bottom of the sealed box 1 by bolts. The first bracket and the second bracket can be fixed by a right-angle connector and bolts.

[0098] 5. A hyperspectral imaging unit is used to collect spectral images of the seeds to be tested and transmit the data to the main control unit, including a hyperspectral camera 6, a halogen lamp 5 and a mounting bracket.

[0099] The hyperspectral camera 6 may be an InGaAs hyperspectral camera 6 , or other types of hyperspectral cameras 6 , such as a CMOS hyperspectral camera 6 , with a spectral range of 953.5 nm to 1720.23 nm.

[0100] The halogen lamp 5 includes two 50W halogen lamps 5 with a color temperature of 3000K, which provide a stable light source for the hyperspectral camera 6 .

[0101] The mounting bracket is used to fix the hyperspectral camera 6 and the halogen lamp 5 on the second bracket of the fixed frame, and adjust their positions and angles.

[0102] The hyperspectral camera 6 is connected to the main control unit via a data line (such as Camera Link, GigE Vision, etc.), and the halogen lamp 5 is connected to the power supply via a power line, and its switch is controlled by the main control unit. The hyperspectral camera 6 and the halogen lamp 5 are fixed on the turntable of the second bracket through a mounting bracket.

[0103] 6. Visible light imaging unit, used to collect visible light images of the seeds to be tested and transmit the data to the main control unit, including a visible light camera 4, an LED bottom light source 9 and a mounting bracket.

[0104] The visible light camera 4 may be a color CCD camera, or other types of cameras, such as a CMOS camera.

[0105] The LED bottom light source 9 adopts a color temperature of 6500K, or other color temperatures, to provide uniform bottom light illumination for the visible light camera 4 .

[0106] The mounting bracket is used to fix the visible light camera 4 on the second bracket of the fixed frame.

[0107] The visible light camera 4 is connected to the main control unit via a USB interface or other interfaces, such as GigE Vision, etc. The LED bottom light source 9 is connected to the power supply via a power cord and is controlled by the main control unit. The visible light camera 4 is fixed to the turntable of the second bracket via a mounting bracket, and the LED bottom light source 9 is placed on the electronic scale 10.

[0108] 7. A robotic arm 7 is used to grab and place a transparent container 8 containing seeds, a background plate 11, and a calibration plate 12 to realize automated operation. A multi-axis collaborative robot or other types of robotic arms 7, such as a SCARA robot, a Delta robot, etc., can be used, and a suitable model and load capacity can be selected according to actual needs.

[0109] The robot arm 7 includes an end effector, and adopts a vacuum suction cup, or other types of clamps, such as a pneumatic gripper, an electromagnetic suction cup, etc., for grasping the transparent container 8.

[0110] The mechanical arm 7 is connected to the main control unit via a communication interface. The base of the mechanical arm 7 is fixed in the sealed box 1.

[0111] 8. The main control unit, as the control center of the entire system, is responsible for controlling the operation of each hardware component, receiving and processing sensor data, running data processing and quality assessment algorithms, and outputting assessment results. It includes embedded systems, power modules, communication modules, and storage modules.

[0112] The embedded system serves as the core of the main control unit and runs the Linux operating system or other operating systems such as Windows IoT.

[0113] The power module provides stable power for the entire system.

[0114] The communication module is used to communicate with various hardware components. For example, it can communicate with a camera, an electronic scale 10, a robotic arm 7, etc. through interfaces such as USB, RS232, and Ethernet.

[0115] The storage module is used to store programs, data and evaluation results, for example, an SD card, a solid state drive, etc. can be used.

[0116] The software programs running on the main control unit include: Low-level drivers that control the operation of individual hardware components.

[0117] Data acquisition module program, used to collect hyperspectral images, visible light images and weight data.

[0118] The data processing module program is used to preprocess, extract features and fuse features of the collected data.

[0119] The quality assessment module program is used to run machine learning models to perform seed quality assessment.

[0120] The control module program is used to coordinate the operation of each module and realize automatic control.

[0121] User interface for displaying evaluation results, setting parameters, monitoring system status, etc.

[0122] 9. Calibration plate 12, used for spectral calibration and radiation positioning of hyperspectral camera 6. Made of material with known spectral reflectance characteristics, such as polytetrafluoroethylene (PTFE) or other diffuse reflective materials. The size of calibration plate 12 needs to be determined according to the camera field of view and actual needs. Before acquiring the sample image, first acquire the image of calibration plate 12 for subsequent spectral correction.

[0123] 10. Background plate 11, used to reduce external environmental interference when the hyperspectral camera 6 is in use, highlight the identification target and provide a reference size. The size of the background plate 11 is determined according to the camera field of view and actual needs. When collecting sample images, the background plate 11 is placed under or around the sample.

[0124] 11. A 360° camera 13 is provided on the inner top of the sealed box 1 to record the working process.

[0125] The control method of the seed quality assessment device of this embodiment includes the following steps: S1. System initialization: Start the device, the main control unit controls the temperature and humidity control unit to adjust the temperature and humidity in the box to the preset values ​​(for example, temperature 25°C, humidity 60%), and wait for the temperature and humidity to stabilize.

[0126] S2. Sample preparation: The main control unit controls the mechanical arm 7 to place the container containing the seeds to be tested on the electronic scale 10 .

[0127] S2. Weight measurement: The main control unit controls the electronic scale 10 to measure the weight of the seeds to be measured, and sends the weight data to the main control unit.

[0128] S3. Hyperspectral image acquisition: The main control unit controls the robot arm 7 to place the whiteboard, controls the halogen lamp 5 to turn on, and controls the hyperspectral imaging unit to collect the whiteboard image; Place the seeds on the whiteboard, control the hyperspectral imaging unit to collect the spectral image of the seeds to be tested, and adjust the exposure time, scanning speed and other parameters of the hyperspectral camera 6 to ensure that the collected spectral image is clear and distortion-free; The main control unit controls the robot arm 7 to remove the seeds and the whiteboard and place a standard blackboard, turn off the halogen lamp 5, and collect the blackboard image. After the collection is completed, the spectral image is sent to the main control unit for black and white correction.

[0129] S8. Visible light image acquisition: The main control unit controls the visible light imaging unit to collect visible light images of the seeds to be tested, controls the bottom LED light source 9 to turn on, and adjusts the exposure time, white balance and other parameters of the visible light camera 4 to ensure that the collected visible light images are clear and the colors are restored realistically. After the collection is completed, the visible light images are sent to the main control unit.

[0130] S9. Data processing and analysis: The main control unit receives the weight data, spectral image and visible light image, and processes and analyzes them according to the aforementioned data processing and quality assessment method to obtain the moisture content, category and quality grade of the seeds.

[0131] S10. Result output: The main control unit outputs the evaluation results (moisture content, category, quality grade) through display screen, network transmission or printer printing.

[0132] S11, Cycle detection: Remove the tested seeds and place the next seed container to be tested, repeat steps 2-10 to achieve continuous testing of multiple samples.

[0133] Example 2 like Figures 1 to 6 As shown, this embodiment provides a seed quality assessment system, device and control method based on multimodal data fusion, which are used to quickly, accurately and non-destructively assess the quality of wheat seeds.

[0134] The seed quality assessment system of this embodiment includes a data acquisition module, a data processing module, a quality assessment module and a control module.

[0135] The data acquisition module includes a hyperspectral imaging unit, a visible light imaging unit, and a weight sensor; The hyperspectral imaging unit adopts a push-broom hyperspectral imaging system, including a CMOS hyperspectral camera 6 (spectral range 400nm to 1000nm, spectral resolution better than 10nm) and a halogen lamp 5. The LED light source is arranged on both sides of the camera, forming an angle of 30 degrees with the camera.

[0136] The visible light imaging unit uses a CMOS camera and a white LED coaxial light source.

[0137] The weight sensor adopts a large-range high-precision electronic scale 10 (the range is 0-2000g and the accuracy is 0.01g).

[0138] The data processing module includes a hyperspectral data processing unit, a visible light image processing unit, and a feature fusion unit. The functions of these units are implemented through software in the industrial computer (the software is written in Python and uses open source libraries such as OpenCV, scikit-learn, and TensorFlow).

[0139] Hyperspectral data processing unit, used to perform the following functions: 1. Black and white correction.

[0140] 2. ROI extraction and average spectrum calculation.

[0141] 3. Characteristic band selection: select 970nm, 1200nm, and 1450nm as characteristic wavelengths.

[0142] 4. Preprocessing: SG smoothing (Savitzky-Golay filter) is used to eliminate noise, the first-order derivative is used to eliminate baseline offset, and multivariate scatter correction (MSC) is used for scatter correction.

[0143] Visible light image processing unit, used to perform the following functions: 1. Image annotation.

[0144] 2. Data enhancement Conditional Generative Adversarial Network (cGAN) is used for data enhancement, especially for categories with a small number of samples such as empty shell kernels, husk kernels and bad kernels. The cGAN structure includes a generator and a discriminator. The generator is used to convert noise and conditional labels (one-hot encoding, dimension 3, representing empty shell kernels, husk kernels and bad kernels, respectively) into images. It consists of a fully connected layer and four convolutional layers. The first four layers use the ReLU activation function, and the output layer uses the Tanh activation function. The discriminator is used to evaluate the authenticity of the image. It consists of four convolutional layers and a fully connected layer. The convolutional layer uses the Leaky ReLU activation function, and the fully connected layer uses the Sigmoid activation function, and the output is a probability between 0 and 1.

[0145] 3. Feature extraction: extract the following visible light image features: Shape characteristics: area, perimeter, aspect ratio, roundness, rectangularity, convexity, etc.

[0146] Color features: mean, standard deviation, skewness, kurtosis, etc. of RGB color components.

[0147] Texture features: energy, contrast, correlation, entropy, etc. extracted by gray-level co-occurrence matrix (GLCM).

[0148] The feature fusion unit performs principal component analysis (PCA) dimensionality reduction on hyperspectral features (spectral values ​​of the selected feature bands), visible light image features (shape features, color features, texture features) and weight data (single seed weight, estimated by dividing the total weight by the number of seeds), retaining 95% of the variance contribution rate (or other proportions, selected according to actual conditions). The reduced-dimensional features are then concatenated to form a comprehensive feature vector.

[0149] The quality assessment module includes a moisture content prediction unit, a seed classification unit and a quality grade assessment unit. The functions of these units are realized through software in the industrial computer.

[0150] The moisture content prediction unit predicts the moisture content of seeds based on the support vector regression (SVR) model. The hyperspectral features of seed samples with known moisture content are used as input and the moisture content is used as output to train the SVR model.

[0151] The seed classification unit classifies seeds based on the YOLOv8 model. It uses annotated visible light images as input (annotation is only required during the early algorithm training) and outputs seed categories (full kernels, empty shell kernels, chaff kernels, impurities, bad kernels) and their location information.

[0152] The quality level assessment unit assesses the quality level of seeds based on the BP neural network model. The structure of the BP neural network is as follows: the number of neurons in the input layer is equal to the dimension of the comprehensive feature vector, the number of neurons in the hidden layer is 20 (or other numbers, selected according to actual conditions), and the number of neurons in the output layer is 3 (corresponding to high, medium, and low quality levels respectively). The comprehensive feature vector of seed samples with known quality levels is used as input, and the quality level is used as output to train the BP neural network model. The Sigmoid activation function is used, and the Adam optimizer is used for training.

[0153] The control module adopts PLC (Programmable Logic Controller) as the control center, and realizes the control and coordination of each module through ladder diagram programming.

[0154] The seed quality assessment device of this embodiment includes: 1. The sealed box 1 is made of aluminum alloy and has shock-absorbing materials inside to reduce the impact of environmental vibration on the device measurement. The dimensions are length × width × height = 1000mm × 800mm × 600mm.

[0155] 2. The temperature and humidity control unit adopts PID control algorithm to improve the temperature and humidity control accuracy. The specific components include temperature sensor 14, humidity sensor 15, fin-type electric heater, TEC semiconductor cooling sheet, industrial ultrasonic humidifier, and compressor dehumidifier.

[0156] 3. An electronic scale 10 with a large range and high precision is used (the range is 0-2000g and the precision is 0.01g).

[0157] The fixed frame is made of industrial plastic material and comprises a first bracket and a second bracket.

[0158] The first bracket adopts a telescopic structure, and the height can be manually adjusted by rotating the handle.

[0159] The second bracket adopts a dovetail groove structure, and the hyperspectral imaging unit and the visible light imaging unit can be fixed by locking screws.

[0160] The first bracket is connected to the sealed box 1 through a buckle.

[0161] The first bracket and the second bracket are fixed by locking screws.

[0162] 4. A hyperspectral imaging unit, comprising a push-broom hyperspectral camera 6, a halogen lamp 5 and a linear displacement platform.

[0163] 5. Push-broom hyperspectral camera 6, corresponding to the description in the data acquisition module.

[0164] 6. Linear displacement platform, which uses a precision linear module driven by a stepper motor with a stroke of 500 mm (or other length) and a repeatability of 0.01 mm, is used to drive the seed sample and pass through the field of view of the hyperspectral camera 6 at a constant speed.

[0165] 7. A visible light imaging unit, comprising a visible light camera 4, a coaxial light source and a mounting bracket. The visible light camera 4 corresponds to the description in the data acquisition module. The coaxial light source is a white LED coaxial light source, which provides uniform illumination for the visible light camera 4.

[0166] 8. The robotic arm 7 adopts a SCARA robot (or other types of robotic arms 7, selected according to actual conditions), and the end effector adopts a pneumatic gripper.

[0167] 9. Main control unit, including industrial computer, power module, communication module, etc. The industrial computer serves as a platform for data processing and quality assessment. The visible light camera 4 is connected to the industrial computer via a USB 3.0 interface. The coaxial light source is connected to the power supply via a power cord, and its switch is controlled by a PLC. The PLC is used as a control center and uses the Siemens S7-1200 series. The industrial computer is connected to the PLC via Ethernet. The industrial computer is connected to the hyperspectral camera 6 and the visible light camera 4 via corresponding interfaces. The PLC is connected to the temperature and humidity control unit, the electronic scale 10, the linear displacement platform, and the robotic arm 7 via corresponding interfaces.

[0168] The control method of the seed quality assessment device of this embodiment includes the following steps: S1. Start the device, and the PLC controls the temperature and humidity control unit to adjust the temperature and humidity in the box to the preset value (for example, temperature 22°C, humidity 55%), and wait for the temperature and humidity to stabilize. The PLC uses the PID control algorithm to achieve precise control of temperature and humidity.

[0169] S2. Sample preparation.

[0170] S3, PLC controls the electronic scale 10 to measure the total weight of the seeds to be tested, and sends the weight data to the industrial computer. The industrial computer calculates the average weight of a single seed according to the preset number of seeds (or obtained by automatic counting of visible light images).

[0171] S4, PLC controls the linear displacement platform to drive the container containing seeds to pass through the field of view of the hyperspectral camera 6 at a constant speed, and the industrial computer controls the hyperspectral camera 6 and the halogen lamp 5 to collect data synchronously. The halogen lamp 5 emits light in sequence according to the preset wavelength sequence, and the hyperspectral camera 6 collects an image at each wavelength, and finally obtains a set of multispectral images.

[0172] S5. PLC controls the visible light imaging unit to collect visible light images of the seeds to be tested and controls the coaxial light source to turn on.

[0173] S6. The industrial computer receives weight data, multispectral images and visible light images, processes and analyzes the data, and performs black and white correction. It extracts ROIs and calculates the average spectrum of each ROI, performs SG smoothing and denoising, first-order derivative elimination of baseline offset, MSC scattering correction, and selects characteristic bands of 970nm, 1200nm, and 1450nm.

[0174] The threshold segmentation algorithm is used to separate the seeds from the background.

[0175] The shape, color and texture features of the seed image after segmentation are extracted, and the seed is segmented from the background using the threshold segmentation algorithm.

[0176] The shape features (area, perimeter, aspect ratio, roundness, rectangularity, convexity, etc.), color features (mean, standard deviation, skewness, kurtosis, etc. of RGB color components) and texture features (energy, contrast, correlation, entropy, etc. extracted from gray-level co-occurrence matrix (GLCM)) of the segmented seed image are extracted.

[0177] The hyperspectral features (spectral values ​​of the selected characteristic bands), visible light image features (shape features, color features, texture features) and weight data (single seed weight) were subjected to PCA dimensionality reduction, retaining 95% of the variance contribution rate. The reduced dimensionality features were then concatenated to form a comprehensive feature vector.

[0178] S7. Use the trained SVR model to predict moisture content.

[0179] S8. Use the trained YOLOv8 model to classify seeds and output category and location information.

[0180] S9. Use the trained BP neural network model to evaluate the quality level.

[0181] S10. The industrial computer displays the evaluation results (moisture content, category, quality grade) on the display and can transmit them to the host computer or database through the network.

[0182] S11, PLC controls the robot arm 7 to remove the tested seed container and place the next seed container to be tested, and repeats steps 5-10 to achieve continuous testing of multiple samples.

[0183] Unless otherwise specifically stated, the relative arrangement, numerical expressions and numerical values ​​of the parts and steps set forth in these embodiments do not limit the scope of the application. In all examples shown and discussed here, any specific value should be interpreted as being merely exemplary, rather than as a restriction. Therefore, other examples of exemplary embodiments can have different values. It should be noted that similar reference numerals and letters represent similar items in the following drawings, and therefore, once a certain item is defined in an accompanying drawing, it does not need to be further discussed in the subsequent accompanying drawings.

[0184] In addition, it should be noted that the use of terms such as "first" and "second" to limit is only for the convenience of distinction. Unless otherwise stated, the above terms have no special meaning and therefore cannot be understood as limiting the scope of protection of this application.

[0185] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A seed quality assessment system based on multimodal data fusion, characterized in that: include: A data acquisition module, used for acquiring multimodal data of seeds, wherein the multimodal data includes a spectral image of the seeds, a visible light image of the seeds, and weight data of the seeds; A data processing module, used for preprocessing, feature extraction and feature fusion of the multimodal data to generate a comprehensive feature vector; A quality assessment module, using a machine learning model to perform seed quality assessment based on the comprehensive feature vector to obtain a seed quality assessment result; A control module, used to control the operation of the data acquisition module, the data processing module and the quality assessment module, and output the seed quality assessment result; The preprocessing includes performing black and white correction, filtering and scattering correction on the spectral image, and marking and data enhancement for target detection on the visible light image; The data processing module comprises: A hyperspectral data processing unit, used for performing black-and-white correction, Savitzky-Golay filtering and Multiplicative Scatter Correction on the spectral image, and outputting hyperspectral features; A visible light image processing unit, used for labeling and data enhancement of the visible light image for target detection, and outputting visible light image features, wherein the data enhancement adopts a conditional generative adversarial network cGAN; The feature fusion unit is used to fuse the hyperspectral feature, the visible light image feature and the weight data to generate the comprehensive feature vector.

2. The seed quality assessment system based on multimodal data fusion according to claim 1, characterized in that: The quality assessment module comprises: A moisture content prediction unit, using a support vector machine model to predict the moisture content of seeds based on the hyperspectral features; A seed classification unit, using a YOLOv8 model, classifies seeds based on the visible light image features, and the seed categories include full grains, empty shell grains, barren grains, impurities, and bad grains; The quality grade evaluation unit uses a random forest model to evaluate the quality grade of the seeds based on the comprehensive feature vector.

3. The seed quality assessment system based on multimodal data fusion according to claim 1, characterized in that: The data acquisition module comprises: A hyperspectral imaging unit, used for collecting spectral images of seeds, wherein the spectral range collected by the hyperspectral imaging unit is 953.5nm to 1720.23nm; A visible light imaging unit, used for collecting visible light images of seeds; Weight sensor, used to collect seed weight data.

4. The seed quality assessment system based on multimodal data fusion according to claim 2, characterized in that: The YOLOv8 model uses GIoU Loss as the loss function, uses NMS for post-processing, uses L1 regularization for parameter pruning, and uses knowledge distillation for model lightweighting; The support vector machine model adopts RBF kernel function and uses Bayesian optimization for parameter tuning; The random forest model outputs the final classification result through a voting mechanism.

5. A seed quality assessment device based on multimodal fusion, used to implement the seed quality assessment system based on multimodal data fusion as described in any one of claims 1 to 4, characterized in that: include: Sealed box; A temperature and humidity control unit, used to control the temperature and humidity in the sealed box; An electronic scale, used for placing the seeds to be tested and measuring the weight of the seeds to be tested; A hyperspectral imaging unit, comprising a hyperspectral camera and a halogen lamp, wherein the hyperspectral camera is used to collect spectral images of the seeds to be tested, and the halogen lamp is used to provide illumination; A visible light imaging unit, comprising a visible light camera and a bottom light source, wherein the visible light camera is used to collect visible light images of the seeds to be tested, and the bottom light source is placed on the electronic scale; A mechanical arm, used for grabbing and placing a container for holding the seeds to be tested; A main control unit, including the data processing module, the quality assessment module and the control module, is communicatively connected with the temperature and humidity control unit, the hyperspectral imaging unit, the visible light imaging unit, the electronic scale and the robotic arm; The temperature and humidity control unit, the hyperspectral imaging unit, the visible light imaging unit, the electronic scale and the mechanical arm are all arranged in the closed box.

6. The seed quality assessment device based on multimodal fusion according to claim 5, characterized in that: It also includes a fixed frame, which is arranged in the closed box; The fixed frame includes a first bracket arranged vertically, and the first bracket is connected to the closed box; The first bracket is also provided with a second bracket arranged horizontally; The hyperspectral imaging unit and the visible light camera are arranged on the second bracket.

7. The seed quality assessment device based on multimodal fusion according to claim 6, characterized in that: A turntable is provided on the second bracket, the hyperspectral camera and the visible light camera are fixed on the turntable, and the halogen lamp is fixed on a side of the hyperspectral camera.

8. A control method for a seed quality assessment device based on multimodal fusion, used to control the seed quality assessment device based on multimodal fusion as claimed in any one of claims 5 to 7, characterized in that: The following steps are involved: Controlling the temperature and humidity control unit to adjust the temperature and humidity in the sealed box to preset values; Controlling the mechanical arm to place the container containing the seeds to be tested on the electronic scale; Controlling the electronic scale to measure the weight data of the seeds to be tested, and sending the weight data to the main control unit; Controlling the hyperspectral imaging unit to collect the spectral image of the seed to be tested, and sending the spectral image to the main control unit; Controlling the visible light imaging unit to collect the visible light image of the seed to be tested, and sending the visible light image to the main control unit; The main control unit receives the weight data, the spectral image and the visible light image, and processes and analyzes the weight data, the spectral image and the visible light image to obtain a seed quality assessment result; The main control unit outputs the seed quality assessment result.

9. The control method of the seed quality assessment device based on multimodal fusion according to claim 8, characterized in that: The step of processing and analyzing the weight data, the spectral image and the visible light image to obtain a seed quality assessment result comprises the following steps: Performing black-and-white correction, Savitzky-Golay filtering and Multiplicative ScatterCorrection scattering correction on the spectral image to obtain a hyperspectral feature; Performing data enhancement on the visible light image, wherein the data enhancement adopts a conditional generative adversarial network cGAN, and then performing target detection annotation to obtain visible light image features; Fusing the hyperspectral features, visible light image features and weight data to generate a comprehensive feature vector; Using a support vector machine model, predicting the moisture content of seeds based on the hyperspectral features; Using the YOLOv8 model, the seeds are classified based on the visible light image features, and the seed categories include full grains, empty shell grains, barren grains, impurities and bad grains; The quality grade of the seeds is evaluated based on the comprehensive feature vector using a random forest model.

Images