A fruit damage class and region recognition method based on multi-source images
By fusing RGB and IR images and combining the loss functions Dice and Focal_loss, the problem of inaccurate fruit damage category and region identification in existing technologies is solved, thus achieving both accuracy and efficiency in fruit damage detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ANHUI UNIV
- Filing Date
- 2023-11-22
- Publication Date
- 2026-06-02
Smart Images
Figure CN117576466B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of fruit damage recognition technology, and in particular to a method for identifying fruit damage categories and regions based on multi-source images. Background Technology
[0002] Fruit damage typically includes pests, diseases, bruises, scratches, and rot. This damage can affect the quality, taste, edibility, and commercial value of the fruit, and may even spread to other healthy apples. Therefore, early identification and management of this damage is crucial for farmers and food producers. Furthermore, fruit damage can impair the appearance, composition, and nutritional value of the fruit, damaging its quality, impacting the profitability of fruit trade, and causing food safety issues. Damage detection can not only stop the spread of damage in time but also improve fruit quality, ensure food safety, and increase profits. Therefore, damage detection of fruit during storage is highly valuable.
[0003] Traditionally, fruit damage detection has relied on manual inspection, a time-consuming, labor-intensive, and error-prone method. Farmers and producers require significant human resources to inspect and sort fruit, potentially leading to inefficiency and increased costs. Furthermore, while other methods for fruit damage detection include spectral analysis and computer vision, these may be ineffective for detecting subtle or difficult-to-distinguish damage. Summary of the Invention
[0004] To address the problem that existing technologies cannot accurately determine the type and area of damage to plant fruits, the present invention aims to provide a method for identifying the type and area of fruit damage based on multi-source images. This method combines RGB and infrared images to detect fruit damage, complementing each other's characteristic information and improving the accuracy of fruit damage detection.
[0005] To achieve the above objectives, the present invention adopts the following technical solution: a method for identifying fruit damage categories and regions based on multi-source images, the method comprising the following sequential steps:
[0006] (1) Construct a visible light and IR imaging system to acquire images of plant fruits under visible light and different infrared wavelengths;
[0007] (2) Fruit images were acquired and processed using visible light and IR imaging systems to obtain a multimodal image dataset, which was then divided into a training set, a validation set, and a test set.
[0008] (3) Improve the DeepLabv3 deep learning model to obtain the ADS-DeepLabv3+ network model;
[0009] (4) The ADS-DeepLabv3+ network model is trained using the training set to obtain the trained ADS-DeepLabv3+ network model. The fruit image to be identified is input into the trained ADS-DeepLabv3+ network model to obtain the fruit damage type and region.
[0010] The step (1) specifically refers to: the visible light and IR imaging system includes a base, a lifting and rotating device is set at the center of the base, the fruit to be identified is placed on the lifting platform of the lifting and rotating device, two first brackets and one second bracket are installed on the base, two lights are fixed on the first brackets respectively, the camera is fixed on the second bracket, a visible light filter is installed in front of the lens of the camera, and the output end of the camera is connected to the input end of the computer.
[0011] Step (2) specifically includes the following steps:
[0012] (2a) Image acquisition:
[0013] The camera is a multispectral camera used to capture visible light images, i.e., RGB images, as well as IR images of different wavelengths;
[0014] (2b) Perform image registration:
[0015] Select a stationary image: Select an RGB image as the stationary image and an IR image as the moving image to be aligned;
[0016] Feature extraction: Extracting feature points or feature regions from the image to be registered;
[0017] Feature matching: Using any one of the scale-invariant feature transformation algorithm, accelerated robust feature algorithm, or fast feature point extraction and description algorithm, the extracted feature points or feature regions are matched to find their correspondence in different images;
[0018] Calculate affine transformation parameters: Based on the matched feature points or feature regions, calculate the affine transformation parameters by minimizing the error between the matched points, including translation vector, rotation angle, scaling ratio, and shear parameter;
[0019] Applying affine transformation: Using the calculated affine transformation parameters, the moving image is transformed to align with the fixed image. Through affine transformation, the position, rotation angle, size, and shape of the image are adjusted to achieve image registration.
[0020] Step (3) specifically includes the following steps:
[0021] (3a1) ResNet is used as the backbone network. A feature pyramid is added to the encoder of the ResNet backbone network, and a feature fusion branch is designed in each level of the feature pyramid to fuse feature maps of different levels and capture multi-scale semantic information.
[0022] (3a2) A convolutional attention module is introduced between the ResNet backbone network and the ASPP module by combining the high-level feature map generated by the ASPP module with the multi-scale feature map generated by the fusion of FPN;
[0023] (3b) Combining the Dice_Loss and Focal_loss loss functions, the loss function of the ADS-DeepLabv3+ network model is improved to obtain a hybrid loss function:
[0024] The Dice_Loss loss function is used to mitigate the impact of imbalance between positive and negative samples. The expression for this function is as follows:
[0025]
[0026] Where TP, FP, and FN represent the values of true positives, false positives, and false negatives, respectively;
[0027] The Focal_loss function is used to adjust the loss between easily distinguishable and difficult-to-distinguish samples, and its expression is as follows:
[0028]
[0029] Where n is the number of classes in the sample, α i It is the weight of the category, p i It is the probability of being predicted as the class, y i γ is the one-hot label of the sample, and γ is a hyperparameter used to adjust the weights of easy and difficult samples;
[0030] The mixed loss function is expressed as follows:
[0031] Loss=Dice_Loss+Focal_Loss.
[0032] As can be seen from the above technical solution, the beneficial effects of the present invention are as follows: First, the present invention can better obtain the "reflection-transmission" characteristics of the fruit, improve the detection accuracy, and have better adaptability and anti-interference ability; Second, the ADS-DeepLabv3+ network model improves the recognition ability of small target damage by incorporating a feature pyramid network and a convolutional attention module, and the hybrid loss function obtained by combining the loss functions of Dice_Loss and Focal_loss improves the prediction ability of difficult-to-distinguish samples. Attached Figure Description
[0033] Figure 1 This is a schematic diagram of the structure of a visible light and IR imaging system;
[0034] Figure 2 This is a schematic diagram of the structure of the ADS-DeepLabv3+ network model in this invention;
[0035] Figure 3 This is an illustration of an Apple defect. Detailed Implementation
[0036] A method for identifying fruit damage categories and regions based on multi-source images, comprising the following sequential steps:
[0037] (1) Construct a visible light and IR imaging system to acquire images of plant fruits under visible light and different infrared wavelengths;
[0038] (2) Fruit images were acquired and processed using visible light and IR imaging systems to obtain a multimodal image dataset, which was then divided into a training set, a validation set, and a test set.
[0039] (3) Improve the DeepLabv3 deep learning model to obtain the ADS-DeepLabv3+ network model;
[0040] (4) The ADS-DeepLabv3+ network model is trained using the training set to obtain the trained ADS-DeepLabv3+ network model. The fruit image to be identified is input into the trained ADS-DeepLabv3+ network model to obtain the fruit damage type and region.
[0041] like Figure 1 As shown, step (1) specifically refers to: the visible light and IR imaging system includes a base 7, a lifting and rotating device is set at the center of the base 7, the fruit to be identified is placed on the lifting platform 3 of the lifting and rotating device, two first supports 5 and one second support 6 are installed on the base 7, two lighting lamps 2 are respectively fixed on the first supports 5, the camera 1 is fixed on the second support 6, a visible light filter is installed in front of the lens of the camera 1, and the output end of the camera 1 is connected to the input end of the computer 4. The camera 1 uses an imaging module with a certain wavelength range. The visible light filter is located in front of the lens and is used for visible light imaging, while the camera 1 with a narrowband filter with a certain center wavelength is used for IR imaging. The computer 4 is equipped with an Intel Celeron Processor J1800 processor.
[0042] Step (2) specifically includes the following steps:
[0043] (2a) Image acquisition:
[0044] The camera 1 is a multispectral camera 1, used to capture visible light images, i.e., RGB images, as well as IR images of different wavelengths;
[0045] (2b) Perform image registration:
[0046] Select a stationary image: Select an RGB image as the stationary image and an IR image as the moving image to be aligned;
[0047] Feature extraction: Extracting feature points or feature regions from the image to be registered;
[0048] Feature matching: Using any one of the scale-invariant feature transformation algorithm, accelerated robust feature algorithm, or fast feature point extraction and description algorithm, the extracted feature points or feature regions are matched to find their correspondence in different images;
[0049] Calculate affine transformation parameters: Based on the matched feature points or feature regions, calculate the affine transformation parameters by minimizing the error between the matched points, including translation vector, rotation angle, scaling ratio, and shear parameter;
[0050] Applying affine transformation: Using the calculated affine transformation parameters, the moving image is transformed to align with the fixed image. Through affine transformation, the position, rotation angle, size, and shape of the image are adjusted to achieve image registration.
[0051] Images from different times, spaces, and modalities require image registration to ensure accurate information correspondence. First, an image is selected as the stationary image, typically a high-quality or high-resolution one. Here, the RGB image is chosen as the stationary image, while the IR image is considered the moving image to be aligned. Next, an affine transformation is used to adjust the moving image to align it with the stationary image. Affine transformations include operations such as translation, rotation, scaling, and shearing, which can adjust the image's position, shape, and size. Using affine transformation parameters, the moving image is transformed to align with the stationary image.
[0052] like Figure 2 As shown, step (3) specifically includes the following steps:
[0053] (3a1) ResNet is used as the backbone network. A feature pyramid is added to the encoder of the ResNet backbone network, and a feature fusion branch is designed in each level of the feature pyramid to fuse feature maps of different levels and capture multi-scale semantic information.
[0054] (3a2) A convolutional attention module is introduced between the ResNet backbone network and the ASPP module to improve perceptual and generalization performance. By combining the high-level feature map generated by the ASPP module with the multi-scale feature map generated by FPN fusion, more accurate pixel-level semantic segmentation is achieved.
[0055] (3b) The defect samples exhibit an imbalance problem, and the difficulty of identifying different defects varies. To address this issue, the loss function of the ADS-DeepLabv3+ network model is improved by combining the Dice_Loss and Focal_loss loss functions, resulting in a hybrid loss function:
[0056] The Dice_Loss loss function is used to mitigate the impact of imbalance between positive and negative samples. The expression for this function is as follows:
[0057]
[0058] Where TP, FP, and FN represent the values of true positives, false positives, and false negatives, respectively;
[0059] The Focal_loss function is used to adjust the loss between easily distinguishable and difficult-to-distinguish samples, and its expression is as follows:
[0060]
[0061] Where n is the number of classes in the sample, α i It is the weight of the category, p i It is the probability of being predicted as the class, y i γ is the one-hot label of the sample, and γ is a hyperparameter used to adjust the weights of easy and difficult samples;
[0062] The mixed loss function is expressed as follows:
[0063] Loss=Dice_Loss+Focal_Loss.
[0064] Example 1:
[0065] 1. Samples and Experiments
[0066] The apple samples were Fuji apples from Zhaoyuan, Yantai City, Shandong Province, China, including apples with a diameter ranging from 70mm to 85mm. Four common defects were identified: black spot, insect damage, bruising, and scratches. Defects, stems, and calyxes were labeled using LabelMe software and formatted into the VOC 2007 dataset format, which includes 121,388 stem and calyx pixels, 133,730 black spot pixels, 52,929 insect damage pixels, 1,031,658 bruise pixels, and 14,656 scratch pixels.
[0067] Dataset Splitting: The augmented dataset was randomly split into three subsets: training, validation, and test. The split ratio was 3:1:1, meaning approximately 60% of the data went to the training set, 20% to the validation set, and 20% to the test set. The training set was used to train the model, the validation set was used to fine-tune hyperparameters, and the test set was used to evaluate the model's performance. Training Parameters: The model was trained for a total of 1000 epochs. One epoch represents one complete iteration through the entire training dataset. Input data was processed in batches, with each batch containing 8 samples. Learning Rate: The initial learning rate was set to 0.0005. Image Size: Input images used for training and evaluation were resized to a resolution of 1080×1080 pixels. Dataset Format: The dataset format used was based on the VOC 2007 dataset format.
[0068] 2. Segmentation effect evaluation indicators
[0069] To evaluate semantic segmentation performance, commonly used metrics were employed: precision (P), recall (R), intersection-over-union (IoU), pixel accuracy (PA), mean intersection-over-union (mIoU), and mean pixel accuracy (mPA). The calculation methods for each evaluation metric are as follows:
[0070]
[0071]
[0072]
[0073]
[0074]
[0075] Where, p ii p represents the total number of pixels belonging to category i and correctly predicted as category i. ij p represents the total number of pixels belonging to category i but incorrectly predicted as j. ji k+1 represents the total number of pixels that were incorrectly predicted as category i but actually belong to category j, and k+1 represents the total number of categories.
[0076] Experimental content and results analysis (results are shown in Table 1):
[0077] The experiment uses apples as an example to analyze fruit damage. The appearance characteristics of apple damage under different wavelength images are as follows: Figure 3As shown. Apple defects include black spot, insect damage, bruising, and scratches, each with different characteristics. Early black spot is difficult to distinguish from the apple's texture, while later it turns dark brown. Insect-damaged areas vary in size and resemble the shape of the fruit stem. Bruises are not very noticeable in RGB images, and scratches are narrow and difficult to detect, sometimes overlapping with other defects. IR images focus more on structural and morphological changes, better displaying bruises and scratches, with 950nm IR images showing the best display effect for bruises and scratches. Based on the above observations, this invention will fuse RGB images and 950nm IR images for subsequent fruit loss detection.
[0078] Apple defect segmentation was performed by combining DeepLabv3+ and ADS-Deeplabv3+ with RGB images. ADS-Deeplabv3+ improved mIoU and mPA by 3.8% and 3.81% compared to DeepLabv3+, respectively. For black spot and insect damage, both IoU and R were significantly improved. For bruises and scratches, IoU and R were improved by 10.94% and 15.37%, respectively, with scratches showing a further 3.26% improvement in IoU, and both R and P values showed slight improvements. Apple defect segmentation was also performed by combining DeepLabv3+ and ADS-Deeplabv3+ with RGB-IR images. ADS-Deeplabv3+ improved mIoU and mPA by 6.51% and 4.89% compared to DeepLabv3+, respectively. Significant improvements were seen in IoU for black spot, insect damage, and scratches, with a slight improvement for bruises, and both R and P values also increased. In summary, combining multi-source images (fusion of RGB and IR images) with ADS-Deeplabv3+ can provide accurate and reliable fruit disease region segmentation results.
[0079] Table 1
[0080]
[0081]
[0082] In summary, this invention can better acquire the "reflection-transmission" characteristics of fruits, improve detection accuracy, and has better adaptability and anti-interference ability. The ADS-DeepLabv3+ network model improves the recognition ability of small target damage by incorporating a feature pyramid network and a convolutional attention module. The hybrid loss function obtained by combining the loss functions Dice_Loss and Focal_loss improves the prediction ability of difficult-to-distinguish samples.
Claims
1. A method for identifying fruit damage categories and regions based on multi-source images, characterized in that: The method includes the following steps in sequence: (1) Construct a visible light and IR imaging system to acquire images of plant fruits under visible light and different infrared wavelengths; (2) Fruit images were acquired and processed using visible light and IR imaging systems to obtain a multimodal image dataset, which was then divided into a training set, a validation set, and a test set. (3) Improve the DeepLabv3 deep learning model to obtain the ADS-DeepLabv3+ network model; (4) The ADS-DeepLabv3+ network model is trained using the training set to obtain the trained ADS-DeepLabv3+ network model. The fruit image to be identified is input into the trained ADS-DeepLabv3+ network model to obtain the fruit damage type and region. Step (3) specifically includes the following steps: (3a1) ResNet is used as the backbone network. A feature pyramid is added to the encoder of the ResNet backbone network, and a feature fusion branch is designed in each level of the feature pyramid to fuse feature maps of different levels and capture multi-scale semantic information. (3a2) A convolutional attention module is introduced between the ResNet backbone network and the ASPP module, by combining the high-level feature map generated by the ASPP module with the multi-scale feature map generated by the fusion of FPN; (3b) Combining the Dice_Loss and Focal_loss loss functions, the loss function of the ADS-DeepLabv3+ network model is improved to obtain a hybrid loss function: The Dice_Loss loss function is used to mitigate the impact of imbalance between positive and negative samples. The expression for this function is as follows: ； Where TP, FP, and FN represent the values of true positives, false positives, and false negatives, respectively; The Focal_loss function is used to adjust the loss between easily distinguishable and difficult-to-distinguish samples, and its expression is as follows: ； Where n is the number of classes in the sample, It is the weight of the category. It is the probability of being predicted as a category. It is the unique hot label of the sample. It is a hyperparameter used to adjust the weights of easy and difficult samples; The mixed loss function is expressed as follows: 。 2. The method for identifying fruit damage categories and regions based on multi-source images according to claim 1, characterized in that: The step (1) specifically refers to: the visible light and IR imaging system includes a base, a lifting and rotating device is set at the center of the base, the fruit to be identified is placed on the lifting platform of the lifting and rotating device, two first brackets and one second bracket are installed on the base, two lights are fixed on the first brackets respectively, the camera is fixed on the second bracket, a visible light filter is installed in front of the lens of the camera, and the output end of the camera is connected to the input end of the computer.
3. The method for identifying fruit damage categories and regions based on multi-source images according to claim 1, characterized in that: Step (2) specifically includes the following steps: (2a) Image acquisition: The camera uses a multispectral camera to capture visible light images, i.e., RGB images, as well as IR images of different wavelengths; (2b) Perform image registration: Select a stationary image: Select an RGB image as the stationary image and an IR image as the moving image to be aligned; Feature extraction: Extracting feature points or feature regions from the image to be registered; Feature matching: Using any one of the scale-invariant feature transformation algorithm, accelerated robust feature algorithm, or fast feature point extraction and description algorithm, the extracted feature points or feature regions are matched to find their correspondence in different images; Calculate affine transformation parameters: Based on the matched feature points or feature regions, calculate the affine transformation parameters by minimizing the error between the matched points, including translation vector, rotation angle, scaling ratio, and shear parameter; Applying affine transformation: Using the calculated affine transformation parameters, the moving image is transformed to align with the fixed image. Through affine transformation, the position, rotation angle, size, and shape of the image are adjusted to achieve image registration.