Multi-spectral imaging on-line identification system for slight damage of fruit surface

This fruit surface minor damage detection system, which combines a multispectral imaging system and a YOLOv8 model, solves the problems of rapid identification and high cost in existing technologies. It enables rapid and accurate detection and classification of minor fruit damage, is applicable to a variety of fruit types, and has broad applicability and economic advantages.

CN118950488AActive Publication Date: 2024-11-15JIANGSU UNIV
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411047199.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-11-15
Estimated Expiration
2044-08-01

AI Technical Summary

Technical Problem

Existing technologies are insufficient for quickly and accurately identifying minor damage to the surface of fruits, and hyperspectral imaging equipment is costly and complex to process, making it unsuitable for real-time industrial detection.

Method used

A multi-lens multispectral camera was designed by combining a multispectral imaging system with the YOLOv8 model, using a sub-window mirror permutation analysis algorithm and a clustering random jumping frog algorithm to screen wavelengths of damage characteristics, and combining it with an electromagnet array system to realize the automatic detection and classification of fruits.

Benefits of technology

It enables rapid and accurate detection and classification of minor damage to fruits, reduces equipment costs, improves detection efficiency and accuracy, and is applicable to a variety of types of fruits, with wide applicability and economic advantages.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118950488A_ABST
    Figure CN118950488A_ABST
Patent Text Reader

Abstract

The invention discloses a multispectral imaging online recognition system for slight damage to the surface of a fruit. The multispectral imaging online recognition system comprises a fruit support, a track, a conveying mechanism, a material arranging mechanism, a detecting mechanism, a classifying mechanism and a collecting mechanism. The material arranging mechanism comprises two guide panels, the detection mechanism comprises a proximity sensor A and a multispectral camera, the multispectral camera is customized according to damage characteristic wavelength, to-be-detected fruit information detected by the proximity sensor A is transmitted to the control unit, and then working of the active light source and the running speed and tension of the conveying belt are controlled. The multispectral camera acquires a spectral image of the fruit to be detected, and the YOLOv8 model performs damage image recognition; the sorting mechanism comprises a pair of electromagnet array systems, proximity sensors B are arranged on the rear sides of the electromagnet array systems, the electromagnet array systems can receive instructions of the control unit and apply accurate magnetic field force to the fruit trays, and slightly damaged fruits and intact fruits are moved to different sides. According to the invention, slight damage of various fruits can be rapidly and accurately detected.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of fruit sorting and grading, and in particular relates to a multi-spectral imaging online recognition system for slight damage on the surface of fruit. Background Art

[0002] Fruits are rich in ingredients such as minerals, vitamins, fatty acids and dietary fiber, and have a sweet and sour taste, which is favored by consumers. However, during the fruit harvesting process, the fruit goes through many links such as picking, packaging, storage and transportation, and is easily affected by external physical factors such as squeezing, collision, vibration and friction, causing physical damage such as slight dents and surface damage to the fruit. These early damages are difficult to identify with the naked eye in the initial stage, and manual inspection will further increase the possibility of physical damage and consume a lot of manpower during the inspection process. When the damaged fruit is packaged and enters the circulation supply chain, the damaged parts are susceptible to external pathogens, causing the fruit to rot, and even the infection spreads to other healthy fruits, causing a large number of products to be unable to be sold normally, causing economic losses while affecting the brand's reputation and image, bringing long-term negative impacts to the industry. Therefore, rapid and non-destructive identification of early damage on the surface of fruit is of great significance to the development of the fruit industry.

[0003] When fruits are damaged by external physical factors, their peel and pulp tissues will undergo physiological and physical changes, such as the diffusion of epidermal cell tissue fluid, which will change the optical properties of the fruit surface. Compared with traditional manual detection methods, optical detection technology has the advantages of high speed, non-destructiveness, and low labor, and is an important means of online detection of the external quality of fruits and vegetables. Conventional machine vision systems can effectively identify serious damage to fruits, such as rot, scars, and crush injuries. However, ordinary machine vision is difficult to fully detect early minor damage to fruits. Hyperspectral imaging technology can clearly detect subtle damage that is difficult to distinguish with the naked eye and RBG images. At the same time, this technology can also provide rich spatial and spectral information, which helps to better understand the damage of fruits. However, hyperspectral imaging technology has the disadvantages of high equipment cost, complex data processing, and limited detection speed. It is not suitable for real-time fruit damage detection in industry. It is necessary to combine the advantages of multispectral imaging systems that can be applied to the rapid detection of a large number of fruits on industrial production lines to meet actual production needs. Summary of the invention

[0004] In view of the shortcomings in the prior art, the present invention provides a multi-spectral imaging online recognition system for slight damage on the surface of fruit.

[0005] The present invention achieves the above technical objectives through the following technical means.

[0006] A multispectral imaging online recognition system for slight damage on the surface of fruit, comprising:

[0007] A plurality of fruit trays and tracks, each of which comprises a fruit tray and a track, wherein the fruit tray is located on the track and can move horizontally along the track;

[0008] The conveying mechanism comprises a bracket 1, a bracket 3 and a conveying chain group, wherein the conveying chain group comprises a conveying belt and a motor, and a plurality of tracks are fixed on the conveying belt at equal intervals;

[0009] Along the moving direction of the conveyor belt, a material sorting mechanism, a detection mechanism, a classification mechanism and a collection mechanism are arranged in sequence;

[0010] The material sorting mechanism includes guide panels arranged on both sides of the conveyor belt, which are used to guide the transported fruits to be tested to the center line position of the conveyor belt;

[0011] The detection mechanism includes a proximity sensor A, a multispectral camera and an active light source. The information of the fruit to be tested detected by the proximity sensor A is transmitted to the control unit. The control unit controls the operation of the active light source and controls the running speed and tension of the conveyor belt according to the signal of the proximity sensor A, so as to ensure that each fruit to be tested can be captured by the multispectral camera in the best state; the multispectral camera is a multi-lens multispectral camera customized according to the damage characteristic wavelength obtained by screening, and different filters are embedded in multiple lenses, and each lens is equipped with a CMOS image sensor; the multispectral camera obtains the spectral image of the fruit to be tested and sends it to the detection terminal, and the YOLOv8 model in the detection terminal performs damage image recognition;

[0012] The classification mechanism includes a set of electromagnet array systems respectively arranged on both sides of the conveyor belt, and a proximity sensor B is arranged on the rear side of the electromagnet array system; the electromagnet array system is integrated with a controller, which can receive instructions from the control unit and apply precise magnetic field force to the fruit tray to move slightly damaged fruits and intact fruits to different sides;

[0013] The collecting mechanism is specifically two slideways.

[0014] In the above technical solution, the rotating shafts at both ends of the conveyor belt are respectively fixed on bracket one and bracket three, one of the rotating shafts is driven by a motor, and the operation of the motor is controlled by a control unit; a pair of baffles are installed between bracket one and bracket three, the material sorting mechanism and the classification mechanism are arranged on the baffles, the detection mechanism is arranged on bracket two, bracket two is located in the middle position of the conveying mechanism, and the collecting mechanism is arranged on bracket three.

[0015] In the above technical solution, the working principle of the electromagnet array system is: in the initial stage, the electromagnet quickly establishes sufficient magnetic field force by passing a large current to ensure that the fruit tray is firmly attracted and moved; then, the current gradually decreases in a proportional manner to slow down the acceleration of the fruit tray.

[0016] In the above technical solution, the damage characteristic wavelength obtained by screening is specifically as follows: the healthy fruits are sequentially collected with hyperspectral image data in the order of numbering, and after each fruit completes two hyperspectral image collections, the collected images are subjected to black-and-white correction, and the average reflectance spectrum of the healthy area of ​​each fruit is extracted; then, the above healthy fruits are artificially slightly damaged, and two hyperspectral image collections and black-and-white corrections are also performed to extract the average reflectance spectrum of the damaged area of ​​each fruit; the average reflectance spectrum data is used to screen the characteristic wavelength of fruit damage.

[0017] In the above technical scheme, the screening of characteristic wavelengths of fruit damage is specifically: using two types of algorithms, a sub-window mirror permutation analysis algorithm and a clustered random leaping frog algorithm, to analyze the average reflectance spectra of healthy fruits and damaged fruits respectively, and the sub-window mirror permutation analysis algorithm obtains the top three band spectra in COSS value, and the clustered random leaping frog algorithm obtains the top three band spectra in SR value, which are respectively used as the damage characteristic wavelengths screened out by the two types of algorithms.

[0018] In the above technical solution, the sub-window mirror replacement analysis algorithm obtains the top three band spectra of COSS values, specifically:

[0019] Set the number of spectral variables j and the number of repetitions N; assume that there are J spectral sub-datasets containing spectral variables j in the sub-datasets obtained by sampling the average reflectance spectrum of fruits through Monte Carlo in the first step; when establishing the PLS-LDA classification model in the second step, there are J PLS-LDA classification models containing spectral variables j. At this time, the prediction error of the prediction set of J models is called the normal prediction error NPE. In addition, the spectral matrix of the prediction set of J models is mirror-permuted, and one of the j spectral variables is selected as the permuted variable. The error of using the model for prediction again is called the permuted prediction error PPE; in the third step, the average values ​​of the J NPEs and PPEs are calculated, recorded as MNPE and MPPE, respectively. When M When NPE>MPPE, the permuted variable is eliminated as a non-informative or interference variable. When MNPE≤MPPE, the permuted variable is retained. For the retained variables, the distribution between J NPEs and PPEs is tested to see whether it is significant, and the p-value of the retained variable is obtained. The conditional synergy score COSS used by the retained variable is evaluated. For the retained variables with a COSS value less than 1, they are directly discarded. For the retained variables with a COSS value greater than 1 and less than 1.5, they are retained and returned to the mirror permutation. Then a permuted variable is added, and the third step is continued until the COSS values ​​of all retained variables are greater than or equal to 1.5. Finally, all retained variables are sorted in descending order according to the COSS value, and the retained variables with the top three COSS values ​​are the damage characteristic wavelengths.

[0020] In the above technical solution, the mirror replacement is specifically divided into:

[0021] In a certain band, for the average reflectance of a single sample, if it is greater than the average reflectance of the population, then subtract twice the difference between the value and the average reflectance of the population; if it is less than the average reflectance of the population, then add twice the difference between the value and the average reflectance of the population; if it is exactly equal to the average reflectance of the population, then remain unchanged.

[0022] In the above technical solution, the clustered random leaping frog algorithm obtains the top three band spectra of SR values, specifically:

[0023] Set the c, ε and k values; in the first step, randomly select K average reflectance spectra as the initial centroids, calculate the distance from the average reflectance spectrum of each sample (fruit) to each initial centroid, and assign each sample to the group where the nearest initial centroid is located. In the second step, calculate the centroid of each group (the centroid is the mean of the average reflectance of all samples in the group), and then redistribute the average reflectance spectrum of each sample to the group where the centroid closest to the average reflectance spectrum of the sample is located, calculate the centroid of the current group again and redistribute it, and iterate the second step until the number of iterations reaches the preset value c; then, the algorithm randomly selects an initial subset V0 containing Q variables as the starting point. In each iteration, the algorithm randomly generates a number Q* from the normal distribution Norm(Q, hQ), which represents the number of variables in the candidate variable subset V*; then, update the initial subset V0 according to the relationship between Q and Q*; then, generate a random number between 0 and 1, and if the random number is greater than or equal to the preset threshold, accept V* as the new feature subset V N+1 , otherwise, reject V* and retain the variable subset of this iteration; the algorithm also calculates the change in classification accuracy between two adjacent iterations. If the change is less than the preset value ε and the number of iterations is greater than the preset value k, the iteration is stopped. Otherwise, it is still iterated until the number of iterations reaches the preset maximum number of iterations. After the iteration, the selection probability SR values ​​of all variables are arranged in descending order, and the variables with the top three SR values ​​are the damage characteristic wavelengths.

[0024] In the above technical solution, the initial subset V0 is updated according to the relationship between Q and Q*, specifically:

[0025] If Q is equal to Q*, the classification accuracy of V* and V0 is calculated and compared. If the classification accuracy of V* is greater than V0, V* replaces V0. If the classification accuracy of V* is less than or equal to V0, no replacement is performed.

[0026] If Q is less than Q*, then Q*-Q variables associated with the smallest regression coefficient are deleted from V0;

[0027] If Q is greater than Q*, then select QQ* variables in V* that are associated with the largest regression coefficient and add them to V0.

[0028] In the above technical scheme, two groups of damage characteristic wavelengths are screened out by the sub-window mirror permutation analysis algorithm and the cluster random leaping frog algorithm respectively, and the images of each group of damage characteristic wavelengths are fused to obtain a total of N characteristic wavelength images of fruit damage, which are divided into training set and verification set of the YOLOv8 model in a ratio of 7:3, and the ratio of damaged fruit characteristic wavelength images to healthy fruit characteristic wavelength images in the verification set is 1:1.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] (1) The present invention uses the YOLOv8 model to identify damaged images. The training set and the validation set of the YOLOv8 model come from two groups of images with damage characteristic wavelengths screened out by the sub-window mirror replacement analysis algorithm and the cluster random leaping frog algorithm, respectively. The present invention is suitable for detecting slight damage of various types of fruits and can clearly and accurately detect slight damage that cannot be distinguished by the naked eye and RGB images. The hyperspectral imaging technology has the advantages of high spectral resolution and a suitable machine learning algorithm to screen damage characteristic wavelengths and customize a multi-lens multispectral camera. Since the multispectral camera has the advantages of low equipment cost and fast image collection, the multispectral camera is applied to the industrial production line to detect slight damage of fruits in real time, which can meet the commercial requirements of fast, accurate and low-cost detection.

[0031] (2) The system of the present invention can automatically and quickly complete the detection and accurate classification of whether the fruit has slight damage, while ensuring that no additional damage is caused to the vulnerable fruit. This feature is particularly important in practical applications. It can not only effectively improve the efficiency of fruit quality detection, but also avoid secondary damage to the fruit caused by improper human operation. In addition, the system has a wide range of applicability to most types of vulnerable fruits, which enhances its practicality in actual production environments. It is worth mentioning that the hardware equipment used in this system is relatively low in cost. This economic advantage makes the system have a high cost-effectiveness and has huge market potential. It can be widely promoted and applied, thus bringing substantial convenience and benefits to the fruit industry.

[0032] (3) Compared with other related inventions, the significant advantages of the present invention are its high practicality and wide applicability, which makes it easier to be adopted and widely used by various enterprises. It is not only suitable for various types of fragile fruits, but also can operate stably under different environments, which greatly enhances its practical value. The present invention has been strictly scientifically verified to ensure the accuracy and reliability of detection and classification, providing a strong quality guarantee for enterprises. Secondly, the system's efficient processing capability is also one of its outstanding advantages. It can quickly complete the detection and classification tasks of a large number of fruits, effectively improving work efficiency. The present invention adopts advanced algorithms and hardware designs to ensure the efficient and stable operation of the system, while also bringing greater competitive advantages to enterprises. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 This is a schematic diagram of using the cluster random frog leaping algorithm to screen characteristic wavelengths according to the present invention;

[0035] Figure 2 This is a schematic diagram of screening characteristic wavelengths using a sub-window mirror replacement analysis algorithm according to the present invention;

[0036] Figure 3 This is a schematic diagram of the structure of the multi-spectral imaging online recognition system for slight damage on the surface of fruit according to the present invention;

[0037] Figure 4 It is a schematic diagram of the material sorting mechanism of the present invention;

[0038] Figure 5 It is a side view of the electromagnet array system of the present invention;

[0039] Figure 6 A side view of the detection mechanism for identifying fruit damage according to the present invention;

[0040] Figure 7 This is a flow chart of the multi-spectral imaging method for identifying slight damage on the surface of fruit according to the present invention;

[0041] In the figure, 101-fruit tray and track, 102-conveying mechanism, 103-material sorting mechanism, 104-detecting mechanism, 105-classification mechanism, 106-collecting mechanism, 107-guiding panel, 108-proximity sensor A, 109-multi-spectral camera, 110-electromagnet array system, 111-active light source, 112-bracket one, 113-bracket three, 114-bracket two, 115-proximity sensor B. DETAILED DESCRIPTION

[0042] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0043] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0044] Example 1

[0045] In order to find out the characteristic wavelengths of damaged fruits, a hyperspectral imaging system is required to collect hyperspectral image data of fruits. First, the prepared healthy fruits are collected in order of numbering, and after each fruit completes two hyperspectral image collections, the collected images are corrected for black and white, and the average reflectance spectrum of the healthy area of ​​each fruit is extracted; next, the above healthy fruits are slightly damaged artificially, and two hyperspectral image collections and black and white corrections are performed to extract the average reflectance spectrum of each damaged area of ​​the fruit; the average reflectance spectrum data is used to screen the characteristic wavelengths of fruit damage. Specifically:

[0046] After the hyperspectral imaging system parameters are set, cover the camera lens cap and collect hyperspectral image data with a reflectance of 0%, which is recorded as the dark reference R dark , and collect the hyperspectral image data of the total reflection of the light source using a polytetrafluoroethylene reflector with a reflection coefficient of 99%, which is recorded as the white reference R white .

[0047] After the fruit hyperspectral image data is collected, the fruit hyperspectral image data is corrected for black and white using the following formula based on the obtained dark reference and white reference.

[0048]

[0049] Among them, ρxy (i) refers to the reflectance of the fruit at the spatial coordinate (x, y) and spectral wavelength i after correction, R xy (i) refers to the original intensity of the fruit spectrum at the spatial coordinate (x, y) and spectral wavelength i.

[0050] Then, the average reflectance spectrum of the damaged area of ​​the fruit and the average reflectance spectrum of the corresponding healthy area were extracted.

[0051] The subwindow mirror permutation analysis algorithm and the clustered random jumping frog algorithm were used to analyze the average reflectance spectrum of healthy fruits, and the subwindow mirror permutation analysis algorithm and the clustered random jumping frog algorithm were also used to analyze the average reflectance spectrum of damaged fruits. The subwindow mirror permutation analysis algorithm obtained several band spectra with higher COSS values, and the clustered random jumping frog algorithm obtained several band spectra with larger SR values, which were used as the damage characteristic wavelengths screened out by the two algorithms respectively.

[0052] like Figure 2As shown, the sub-window mirror replacement analysis algorithm analyzes the average reflectance spectrum of the fruit and selects several damage characteristic wavelengths in the near-infrared band. Set the number of spectral variables j and the number of repetitions N. Assume that there are J spectral sub-datasets containing spectral variables j in the sub-datasets obtained by sampling the average reflectance spectra of fruits through Monte Carlo in the first step. When the PLS-LDA classification model is established in the second step, there will be J PLS-LDA classification models containing spectral variables j. At this time, the prediction errors of the prediction sets of these J models are called normal prediction errors (Normal Prediction Errors, NPE). In addition, the spectral matrix of the prediction set of the J models is mirror-permuted. The mirror-permutation is specifically divided into: in a certain band, for the average reflectance of a single sample, if it is greater than the average reflectance of the whole, then it is subtracted by twice the difference between the value (that is, the average reflectance of the single sample in this band) and the average reflectance of the whole. If it is less than the average reflectance of the whole, then it is added with twice the difference between the value and the average reflectance of the whole. If it is exactly equal to the average reflectance of the whole, it remains unchanged; select a variable among the j spectral variables as the permuted variable; the error of using the model for prediction again is called the permuted prediction error (Permuted Prediction Error). Errors, PPE); in the third step, the average values ​​of the obtained J NPEs and PPEs are calculated, recorded as MNPE and MPPE. When MNPE>MPPE, the permuted variable is eliminated as a non-informative or interference variable. When MNPE≤MPPE, the permuted variable is retained. For the retained variables (j spectral variables), the Mann-Whitney U method is used to test whether the distribution between the J NPEs and PPEs is significant, and the p-value of the retained variable is obtained. The Conditional Synergetic Score (COSS) used by the retained variable is evaluated, which is the logarithm of the negative p-value (COSS=-lgp); the larger the COSS value, the more important the retained variable; for the retained variables with COSS values ​​less than 1, they are directly discarded; for the retained variables with COSS values ​​greater than 1 and less than 1.5, they are retained and returned to the mirror permutation step, and then a permuted variable is added to perform the third step until the COSS values ​​of all retained variables are greater than or equal to 1.5. Finally, all retained variables are sorted in descending order according to the COSS value, and the retained variables with the largest COSS values ​​are the damage characteristic wavelengths.

[0053] like Figure 1As shown in the figure, in the clustered random leap frog algorithm, the c, ε and k values ​​are set, the average reflectance spectrum of the fruit is analyzed, and several damage characteristic wavelengths are screened out. In the first step, K average reflectance spectra are randomly selected as the initial centroids, the distance from the average reflectance spectrum of each sample (fruit) to each initial centroid is calculated, and each sample is assigned to the group where the nearest initial centroid is located; in the second step, the centroid of each group is calculated (the centroid is the mean of the average reflectance of all samples in the group), and then the average reflectance spectrum of each sample is redistributed to the group where the centroid closest to the average reflectance spectrum of the sample is located, and the centroid of the current group is calculated again and redistributed; the second step is iterated until the number of iterations reaches the preset value c. Next, the algorithm randomly selects an initial subset V0 containing Q variables as the starting point. In each iteration, the algorithm randomly generates a number Q* from the normal distribution Norm(Q, hQ), which represents the number of variables in the candidate variable subset V*. Then, according to the clustering results, a cross-group selection strategy is used to generate the candidate variable subset V*. Next, the algorithm updates the initial subset V0 according to the relationship between Q and Q*: if Q is equal to Q*, the partial least squares discriminant analysis method (PLS-DLA) is used to calculate and compare the classification accuracy of V* and V0. If the classification accuracy of V* is greater than V0, V* replaces V0. If the classification accuracy of V* is less than or equal to V0, no replacement is made. If Q is less than Q*, Q*-Q variables related to the minimum regression coefficient are deleted from V0. If Q is greater than Q*, QQ* variables related to the maximum regression coefficient in V* are selected and added to V0. After that, the algorithm generates a random number between 0 and 1. If the random number is greater than or equal to the preset threshold, V* is accepted as the new feature subset V N+1 ; Otherwise, reject V* and retain the variable subset of this iteration. The algorithm also calculates the change in classification accuracy between two adjacent iterations. If the change is less than the preset value ε and the number of iterations N is greater than the preset value k, the iteration is stopped. Otherwise, the iteration is continued until the number of iterations N reaches the preset maximum number of iterations N Max Finally, the selection probability SR values ​​of all variables are arranged in descending order, and the variables with the largest SR values ​​are the damage characteristic wavelengths.

[0054] Two groups of damage characteristic wavelengths were screened out by the sub-window mirror permutation analysis algorithm and the cluster random jumping frog algorithm respectively. The merge() function in the OpenCV library was used to fuse the images of each group of damage characteristic wavelengths. A total of N characteristic wavelength images of fruit damage were obtained, which were divided into training set and validation set in a ratio of 7:3. The ratio of damaged fruit characteristic wavelength images to healthy fruit characteristic wavelength images in the validation set was 1:1.

[0055] YOLOv8 is a target detection algorithm based on deep learning. It has the advantages of fast speed, light weight, high accuracy, and open source program, and is widely used in the field of industrial detection. The present invention uses the YOLOv8 model to identify the target of slightly damaged fruits, which is built into the detection terminal.

[0056] When training the YOLOv8 model using the training set, first, you need to configure the appropriate virtual environment and start running, then import the fruit characteristic wavelength image, use Labelimg as the image data annotation tool, draw the target rectangular box around the damaged part of the fruit, and enter the category label to complete the annotation. During the annotation process, you need to ensure that the rectangular box fits closely around the target, because a more realistic annotation box can help the model training achieve better expected performance. After the annotation is completed, a annotation file will be generated for each image data. Each row of the annotation file represents a target object in the image. The first column of each row is the category index of the target, and the next few columns represent the location information of the rectangular box. Put the annotated image data into the images folder, and the corresponding annotation file into the labels folder.

[0057] Before model training, you need to configure the environment and adjust the parameters, which include model parameters and hyperparameters. Model parameters include the number of convolution kernels and pooling layers. These parameters are fixed, but can be optimized for the target task later. Hyperparameters include learning rate lr, epoch, batch-size, and loss function. Among them, the batch-size should match the GPU to the maximum, which is set to a1 here; epoch should be set according to the convergence of the loss function and mAP accuracy, which is set to a2 here; the learning rate lr is set to a3.

[0058] When using the validation set for validation, the F1 scores of the models trained on the SMPAA dataset (a set of images corresponding to the characteristic wavelengths of the sub-window mirror permutation analysis algorithm) and the CRF dataset (a set of images corresponding to the characteristic wavelengths of the clustered random leaping frog algorithm) were A1 and A2, respectively.

[0059] Example 2

[0060] This embodiment proposes a multi-spectral imaging online recognition system for slight damage on the surface of fruit, which is used for real-time, rapid, non-destructive and accurate detection and classification. Figure 3 As shown, the system includes a fruit tray and track 101 and a conveying mechanism 102 for transporting the fruits to be tested in batches.

[0061] The fruit tray and track 101 include two parts, namely, the fruit tray and the track. The fruit tray is located on the track and can move horizontally along the track. The fruit tray includes a cup body made of transparent plastic material and a cup bottom containing iron products. The transparent cup body is conducive to the transmission of light and will not reduce the accuracy of model recognition due to blocking the light source. The iron cup bottom is the key to the classification operation of the classification mechanism. The sliding track can move the fruit tray in a specified direction without falling. The sliding track is made of lightweight and high-strength plastic material.

[0062] The conveying mechanism 102 includes a bracket 112, a bracket 3 113 and a conveying chain group, wherein the conveying chain group includes a conveyor belt and a motor, and is used to continuously convey the fruit tray loaded with the fruit to be tested and the track 101; the rotating shafts at both ends of the conveyor belt are respectively fixed on the bracket 112 and the bracket 3 113, one of the rotating shafts is driven by a motor, and the operation of the motor is controlled by a control unit; a pair of baffles are installed between the bracket 112 and the bracket 3 113. The conveyor belt is made of wear-resistant polyurethane (PU) material, the track is fixed to the conveyor belt, and each track is arranged at equal intervals.

[0063] Along the moving direction of the conveyor belt, a material sorting mechanism 103, a detection mechanism 104, a classification mechanism 105 and a collection mechanism 106 are sequentially arranged. The material sorting mechanism 103 and the classification mechanism 105 are arranged on the baffle, the detection mechanism 104 is arranged on the bracket 2 114, the bracket 2 114 is located in the middle of the conveying mechanism 102, and the collection mechanism 106 is arranged on the bracket 3 113.

[0064] The material sorting mechanism 103 includes guide panels 107 ( Figure 4 ), the guide panel 107 is a rubber roller in the shape of a funnel, which is used to guide the fruit to be tested transported on the conveying mechanism 102 to the center line position of the conveyor belt so as to pass through the detection station.

[0065] The detection mechanism 104 is arranged at the detection station, and is used to obtain the spectral image of the fruit to be tested at the detection station in real time, and transmit the spectral image to the detection terminal, which is built in the control unit; the detection terminal detects whether the fruit to be tested has slight damage based on the spectral data, and transmits the result to the classification mechanism 105. When the fruit to be tested passes through the detection station, the proximity sensor A108 detects when the fruit enters and leaves the detection area, as well as the position information of the fruit in the detection area, and transmits it to the control unit. The control unit first controls the active light source 111 to work, and controls the running speed and tension of the conveyor belt according to the signal of the proximity sensor 108, to ensure that each fruit can be captured by the multi-spectral camera 109 in the best state. Through such a design, the conveying mechanism 102 of this embodiment can cooperate closely with the multi-spectral camera 109 to realize the online identification and classification of slight damage on the surface of the fruit, thereby improving the efficiency and accuracy of fruit quality detection. See Figure 5 .

[0066] Optionally, the multispectral camera 109 is a multi-lens multispectral camera customized according to the damage characteristic wavelength screened according to Example 1, and different filters are embedded in multiple lenses to obtain spectral images of different bands. Each lens is equipped with a CMOS image sensor with 300,000 pixels and an integration type of snapshot to support fast and accurate collection of fruit image information during actual detection. Multiple lenses simultaneously shoot the fruit to be tested, and finally transmit the image to the detection terminal via a USB3.0 interface. The multispectral camera 109 has a built-in TEC semiconductor cooler to ensure that the multispectral camera 109 is at a normal operating temperature.

[0067] The classification mechanism 105 classifies the fruits according to whether they are slightly damaged, and the classified fruits slide into the track of the collection mechanism 106. The classification mechanism 105 includes an electromagnet array system 110, and a proximity sensor B115 is arranged on the baffle at the rear side of the electromagnet array system 110; Figure 6 As shown, a set of electromagnet array system 110 is respectively set on both sides of the conveyor belt, and two electromagnet array systems 110 are arranged relatively, and the system is integrated with a controller, can receive the instruction of the control unit and apply accurate magnetic field force to the fruit tray, thereby guiding the fruit tray and the fruit on its upper part to be classified and moved along a predetermined path, and slightly damaged fruits and intact fruits are moved to different sides. In the initial stage, the electromagnet quickly establishes enough magnetic field forces by passing a large current to ensure that the fruit tray is firmly attracted and moved. Subsequently, the current gradually decreases in a proportional manner, and this dynamic adjustment mechanism effectively slows down the acceleration of the fruit tray, avoiding potential damage to the fruit due to excessive instantaneous impact force. In addition, the magnetic force between the iron fruit tray and the electromagnet, combined with the guiding effect of the track, jointly constitutes a stable and reliable classification transmission system. When the classification is completed, the fruit on the fruit tray will roll to the slideway of the collecting mechanism 106 with the tilt of the fruit tray.

[0068] The collecting mechanism 106 is specifically two slideways.

[0069] Example 3

[0070] This example takes the classification of slightly damaged and undamaged strawberries on an industrial assembly line as an example, and adopts the multispectral imaging online recognition system for slight damage on the surface of fruits of the present invention to realize the rapid detection and classification of slight damage on industrial online strawberries, as follows:

[0071] Hyperspectral image data of strawberries were collected using a hyperspectral imaging system (400-1000nm). First, the prepared healthy strawberries were collected in order of numbering, and spectral image data was collected in sequence. After each strawberry completed two hyperspectral image collections, the collected images were corrected for black and white, and the average reflectance spectrum of the healthy area of ​​each strawberry was extracted; next, the healthy strawberries were slightly damaged artificially, and two hyperspectral image collections and black and white corrections were performed to extract the average reflectance spectrum of the damaged area of ​​each strawberry; the average reflectance spectrum data was used to screen the characteristic wavelength of strawberry damage.

[0072] After the strawberry hyperspectral image data is collected, the black and white correction is performed on the strawberry hyperspectral image data based on the dark reference and white reference. Two methods are used to analyze the average reflectance spectra of healthy strawberries and damaged strawberries to screen out the characteristic wavelengths of strawberry damage.

[0073] The sub-window mirror replacement analysis algorithm screened out three damage characteristic wavelengths in the near-infrared band by analyzing the average reflectance spectrum of strawberries. In the 900-1100nm band, the near-infrared spectral characteristics are closely related to moisture, and the damaged parts of the fruit have changes in moisture due to damage to the cell walls and cell membranes of the flesh cells. The spectral bands at 737.1nm and 767.6nm selected by the clustered random leaping frog algorithm are related to the third overtone of the OH group, and 963.1nm is related to moisture, which is also closely related to moisture. Therefore, the state of moisture changes on the surface of the damaged part of the strawberry can be obtained.

[0074] The two groups of damage characteristic wavelengths of 958.6nm, 974nm, 1025.6nm and 737.1nm, 767.6nm and 963.1nm were respectively selected by the sub-window mirror replacement analysis algorithm and the cluster random leap frog algorithm, and each group of damage characteristic wavelength images was image fused using the merge() function in the OpenCV library. This embodiment generates 200 strawberry damage images, which are divided into a training set and a validation set at a ratio of 0.7 and 0.3, and the ratio of the damaged fruit characteristic wavelength images to the healthy fruit characteristic wavelength images in the validation set is 1:1.

[0075] The training hardware of the YOLOv8 model uses the Windows 11 operating system, the internal GPU graphics card is GTX3060tap, the video memory size is 6G, the CPU core is AMD R7 5800H, and the memory is 32G. To run the support files required for YOLOv8, the virtual environment is configured through anaconda, the Python version is 3.8.18, the PyTorch version is 2.1.0, and the CUDA version is 11.8.0. Before model training, the batch-size is set to 8, the epoch is set to 1000, and the learning rate is set to 0.001.

[0076] When validated using an independent validation set, the F1 scores of the models trained on the SMPAA and CRF datasets were 0.9583 and 0.930, respectively.

[0077] According to the optimization results, the wavelengths of 958.6nm, 974nm, and 1025.6nm screened by the sub-window mirror replacement analysis algorithm were selected as the shooting wavelengths of the multispectral camera, and a three-lens multispectral camera was customized according to these three wavelengths.

[0078] The strawberries to be tested are placed in the fruit tray in the horizontal direction behind the material sorting mechanism 103, and the fruit trays on the slide are arranged into a row along the center line of the conveyor belt by the material sorting mechanism 103. When the fruit to be tested passes through the detection station, the proximity sensor A108 in the detection mechanism 104 detects when the fruit tray enters and leaves the detection area, as well as the position information of the fruit tray in the detection area. When the fruit tray passes directly under the multi-spectral camera 109, the proximity sensor A108 sends a signal to the control unit. The control unit first controls the active light source 111 to work, and controls the running speed and tension of the conveyor belt according to the signal of the proximity sensor 108 to ensure that each strawberry is captured by the multi-spectral camera 109 in the best state. The spectral image is transmitted to the detection terminal, and the YOLOv8 model performs damage image recognition (if it is identified that the strawberry on the i-th fruit tray is slightly damaged, the fruit tray when it reaches the area opposite to the two electromagnet array systems 110 is recorded as the i+m-th), and then the detection result is transmitted to the control unit. The control unit controls the classification mechanism 105 to act according to the inspection result, such as Figure 7As shown. In the sorting mechanism 105, the proximity sensor B115 is responsible for monitoring whether the strawberry passes through the relative area of ​​the two electromagnet array systems 110, and sends it to the control unit. In this relative area, the healthy strawberry and the slightly damaged strawberry are respectively activated by the electromagnet array system 110 of the corresponding side according to the signal of the control unit, and the current of the corresponding side is activated. In the initial stage, the electromagnet quickly generates a strong magnetic field force by applying a large current to ensure the effective capture of the fruit tray in front of it. As the fruit tray gradually approaches the electromagnet, the current in the electromagnet coil is gradually reduced in a proportional manner. Finally, when the fruit tray approaches the preset position, the current is reduced to zero, and a smooth transition is achieved. Subsequently, the conveyor belt carries the fruit tray and the classified strawberries thereon and continues to move forward. When the fruit tray reaches a specific tilt angle (i.e., the fruit tray is relative to the slide entrance at this time), the strawberry naturally slides out of the fruit tray under the action of gravity, and is finally properly collected by the special collection panel of the collection mechanism 106.

[0079] The classification results of this embodiment are shown in Table 1:

[0080] Table 1 Classification results of this embodiment

[0081] Classification Total number of samples Number of correct classifications Number of misclassifications Classification accuracy Healthy Strawberries 500 488 12 97.6% Slightly damaged strawberries 500 487 13 97.4% overall 1000 975 25 97.5%

[0082] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0083] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A multispectral imaging online recognition system for slight damage on the surface of fruit, characterized in that: include: A plurality of fruit trays and tracks (101), each of the fruit trays and tracks (101) comprising a fruit tray and a track, the fruit tray being located on the track and being able to move horizontally along the track; The conveying mechanism (102) comprises a bracket 1 (112), a bracket 3 (113) and a conveying chain group, wherein the conveying chain group comprises a conveying belt and a motor, and a plurality of tracks are fixed on the conveying belt at equal intervals; A material sorting mechanism (103), a detection mechanism (104), a classification mechanism (105) and a collection mechanism (106) are sequentially arranged along the moving direction of the conveyor belt; The material sorting mechanism (103) comprises guide panels (107) arranged on both sides of the conveyor belt, and is used to guide the transported fruits to be tested to the center line position of the conveyor belt; The detection mechanism (104) comprises a proximity sensor A (108), a multispectral camera (109) and an active light source (111). The information of the fruit to be tested detected by the proximity sensor A (108) is transmitted to the control unit. The control unit controls the operation of the active light source (111) and controls the running speed and tension of the conveyor belt according to the signal of the proximity sensor A (108), so as to ensure that each fruit to be tested can be captured by the multispectral camera (109) in an optimal state. The multispectral camera (109) is a multi-lens multispectral camera customized according to the damage characteristic wavelength obtained by screening, and different filters are embedded in the multiple lenses, and each lens is equipped with a CMOS image sensor. The multispectral camera (109) obtains the spectral image of the fruit to be tested and sends it to the detection terminal. The YOLOv8 model in the detection terminal performs damage image recognition. The classification mechanism (105) comprises a set of electromagnet array systems (110) respectively arranged on both sides of the conveyor belt, and a proximity sensor B (115) is arranged on the rear side of the electromagnet array system (110); the electromagnet array system (110) is integrated with a controller, which can receive instructions from the control unit and apply precise magnetic field force to the fruit tray to move slightly damaged fruits and intact fruits to different sides; The collecting mechanism (106) is specifically two slideways.

2. The multispectral imaging online identification system for slight damage on the fruit surface according to claim 1, characterized in that: The rotating shafts at both ends of the conveyor belt are respectively fixed on bracket one (112) and bracket three (113), one of the rotating shafts is driven by a motor, and the operation of the motor is controlled by a control unit; a pair of baffles are installed between the bracket one (112) and the bracket three (113), the material sorting mechanism (103) and the classification mechanism (105) are arranged on the baffles, the detection mechanism (104) is arranged on bracket two (114), the bracket two (114) is located in the middle position of the conveying mechanism (102), and the collecting mechanism (106) is arranged on bracket three (113).

3. The multispectral imaging online identification system for slight damage on the fruit surface according to claim 1, characterized in that: The working principle of the electromagnet array system (110) is as follows: in the initial stage, the electromagnet quickly establishes sufficient magnetic field force by passing a large current to ensure that the fruit tray is firmly attracted and moved; Subsequently, the current gradually decreases in a proportional manner, slowing down the acceleration of the fruit tray.

4. The multispectral imaging online identification system for slight damage on the fruit surface according to claim 1, characterized in that: The damage characteristic wavelength obtained by screening is specifically as follows: the healthy fruits are sequentially collected with hyperspectral image data in the order of numbering, and after each fruit completes two hyperspectral image collections, the collected images are subjected to black-and-white correction, and the average reflectance spectrum of the healthy area of ​​each fruit is extracted; then, the healthy fruits are artificially slightly damaged, and two hyperspectral image collections and black-and-white corrections are also performed to extract the average reflectance spectrum of the damaged area of ​​each fruit; the average reflectance spectrum data is used to screen the damage characteristic wavelength of the fruit.

5. The multi-spectral imaging online identification system for slight damage on the fruit surface according to claim 4, characterized in that: The method of screening the characteristic wavelengths of fruit damage specifically comprises: using two types of algorithms, a sub-window mirror permutation analysis algorithm and a cluster random leaping frog algorithm, to analyze the average reflectance spectra of healthy fruits and damaged fruits respectively, the sub-window mirror permutation analysis algorithm obtains the top three band spectra of COSS values, and the cluster random leaping frog algorithm obtains the top three band spectra of SR values, which are respectively used as the damage characteristic wavelengths screened out by the two types of algorithms.

6. The multi-spectral imaging online identification system for slight damage on the fruit surface according to claim 5, characterized in that: The sub-window mirror replacement analysis algorithm obtains the top three band spectra of COSS values, specifically: Set the number of spectral variables j and the number of repetitions N; assume that there are J spectral sub-datasets containing spectral variables j in the sub-datasets obtained by sampling the average reflectance spectrum of fruits through Monte Carlo in the first step; when establishing the PLS-LDA classification model in the second step, there are J PLS-LDA classification models containing spectral variables j. At this time, the prediction error of the prediction set of J models is called the normal prediction error NPE. In addition, the spectral matrix of the prediction set of J models is mirror-permuted, and one of the j spectral variables is selected as the permuted variable. The error of using the model for prediction again is called the permuted prediction error PPE; in the third step, the average values ​​of the J NPEs and PPEs are calculated, recorded as MNPE and MPPE, respectively. When M When NPE>MPPE, the permuted variable is eliminated as a non-informative or interference variable. When MNPE≤MPPE, the permuted variable is retained. For the retained variables, the distribution between J NPEs and PPEs is tested to see whether it is significant, and the p-value of the retained variable is obtained. The conditional synergy score COSS used by the retained variable is evaluated. For the retained variables with a COSS value less than 1, they are directly discarded. For the retained variables with a COSS value greater than 1 and less than 1.5, they are retained and returned to the mirror permutation. Then a permuted variable is added, and the third step is continued until the COSS values ​​of all retained variables are greater than or equal to 1.

5. Finally, all retained variables are sorted in descending order according to the COSS value, and the retained variables with the top three COSS values ​​are the damage characteristic wavelengths.

7. The multi-spectral imaging online identification system for slight damage on the fruit surface according to claim 6, characterized in that: The mirror replacement is specifically divided into: In a certain band, for the average reflectance of a single sample, if it is greater than the average reflectance of the population, then subtract twice the difference between the value and the average reflectance of the population; if it is less than the average reflectance of the population, then add twice the difference between the value and the average reflectance of the population; if it is exactly equal to the average reflectance of the population, then remain unchanged.

8. The multi-spectral imaging online identification system for slight damage on the fruit surface according to claim 5, characterized in that: The clustered random leaping frog algorithm obtains the top three band spectra of SR values, specifically: Set the c, ε and k values; in the first step, randomly select K average reflectance spectra as the initial centroids, calculate the distance from the average reflectance spectrum of each sample to each initial centroid, and assign each sample to the group where the nearest initial centroid is located. In the second step, calculate the centroid of each group, and then redistribute the average reflectance spectrum of each sample to the group where the centroid closest to the average reflectance spectrum of the sample is located, calculate the centroid of the current group again and redistribute it, and iterate the second step until the number of iterations reaches the preset value c; then, the algorithm randomly selects an initial subset V0 containing Q variables as the starting point. In each iteration, the algorithm randomly generates a number Q* from the normal distribution Norm(Q, hQ), which represents the number of variables in the candidate variable subset V*; then, the initial subset V0 is updated according to the relationship between Q and Q*; after that, a random number between 0 and 1 is generated. If the random number is greater than or equal to the preset threshold, V* is accepted as the new feature subset V N+1 , otherwise, reject V* and retain the variable subset of this iteration; the algorithm also calculates the change in classification accuracy between two adjacent iterations. If the change is less than the preset value ε and the number of iterations is greater than the preset value k, the iteration is stopped. Otherwise, it is still iterated until the number of iterations reaches the preset maximum number of iterations. After the iteration, the selection probability SR values ​​of all variables are arranged in descending order, and the variables with the top three SR values ​​are the damage characteristic wavelengths.

9. The multi-spectral imaging online identification system for slight damage on the fruit surface according to claim 8, characterized in that: The initial subset V0 is updated according to the relationship between Q and Q*, specifically: If Q is equal to Q*, the classification accuracy of V* and V0 is calculated and compared. If the classification accuracy of V* is greater than V0, V* replaces V0. If the classification accuracy of V* is less than or equal to V0, no replacement is performed. If Q is less than Q*, then Q*-Q variables associated with the smallest regression coefficient are deleted from V0; If Q is greater than Q*, then select QQ* variables in V* that are associated with the largest regression coefficient and add them to V0.

10. The multi-spectral imaging online identification system for slight damage on the fruit surface according to claim 5, characterized in that: Two groups of damage characteristic wavelengths were screened out by the sub-window mirror permutation analysis algorithm and the clustering random leaping frog algorithm respectively. The images of each group of damage characteristic wavelengths were fused to obtain a total of N characteristic wavelength images of fruit damage. They were divided into the training set and validation set of the YOLOv8 model in a ratio of 7:3, and the ratio of the damaged fruit characteristic wavelength images to the healthy fruit characteristic wavelength images in the validation set was 1:1.

Citation Information

Patent Citations

  • Non-destructive testing device for protein conformation in egg white and method of non-destructive testing device

    CN106018292A

  • Motor with oblique magnetizing magnet

    CN109980809A

  • Apple slight damage classification method

    CN110095436A

  • Method for detecting early recessive damage of Korla pear by fusing hyperspectral image features and spectral variables

    CN117491382A

  • Method for improving precision of LIBS (laser-induced breakdown spectroscopy) detection of content of calcium element in drilling carbonate rock

    CN117609872A