A method of fruit defect detection

Abstract

The application provides a fruit defect detection method, and relates to the technical field of image processing, which comprises the following steps: collecting image data containing fruit defect detection under different conditions, cleaning the image data, generating an initial data set and labeling; pre-processing the labeled image data and dividing it into a training set, a validation set and a test set; constructing an improved YOLOV5 network model and training and verifying it using the training set and the validation set; according to the test set, collecting and pre-processing image data containing fruit defects, using the improved YOLOV5 network model, the model fuses a context guide block and a high-resolution feature pyramid network to enhance feature extraction and fusion, and optimizes a loss function to speed up the convergence of model training, and finally realizes efficient detection of fruit defects such as scars and breakage. The application solves the problem that fruit defects cannot be quickly and accurately detected in the target detection field.

Description

Technical Field

[0001] This invention belongs to the field of fruit defect technology, and in particular relates to a method for detecting fruit defects. Background Technology

[0002] The quality of fruits and vegetables directly affects food safety, which in turn is closely related to people's health and well-being. Defect detection can effectively prevent substandard fruit from entering the market, protecting consumer health. For example, apple defect detection involves examining the surface quality, color, shape, and size of the fruit to identify blemishes or defects. Common apple defects include scars, breakage, rot, deformation, and pest infestation.

[0003] In recent years, deep learning-based object detection methods have been successfully applied in many fields with good detection results. However, in the field of fruit defect detection, apples may occlude or overlap in reality, which makes object detection more difficult. Small object detection, dense leak points, positioning accuracy, and adaptive detection in complex backgrounds and environments are still potential challenges. Summary of the Invention

[0004] In view of the above-mentioned shortcomings in the prior art, the present invention provides a fruit defect detection method, which solves the problem that fruit defects cannot be detected quickly and accurately in the field of target detection.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is: a method for detecting fruit defects, comprising the following steps:

[0006] S1. Collect image data containing fruit defect detection under different conditions, clean the data, generate an initial dataset, and label it;

[0007] S2. Preprocess the labeled image data and divide it into training set, validation set and test set;

[0008] S3. Construct an improved YOLOv5 network model;

[0009] S4. Train and validate the improved YOLOv5 network model using the training and validation sets;

[0010] S5. Based on the test set, use the trained and validated YOLOv5 network model to detect fruit defect images and obtain the detection results.

[0011] Further, S2 includes the following steps:

[0012] S201. Randomly select an image and stitch together the coordinates of the reference points;

[0013] S202. Randomly select four images from the labeled image data, and based on the reference point, adjust the size and position of the four images and place them at four positions in a layer of a specified size, and map the relationship to the image labels.

[0014] S203. By stitching together the images in the layers, the coordinates of the out-of-bounds detection box are cropped;

[0015] S204. Randomly crop a square region of a fixed size from the latitude and longitude of the image and fill the region with zero values ​​to complete the preprocessing of the image data.

[0016] S205. Divide the preprocessed image data into training set, validation set and test set.

[0017] Furthermore, S3 specifically refers to:

[0018] The convolutional module is improved by using the context-guided block module, the feature pyramid network structure is improved into a high-resolution feature pyramid network feature fusion structure, and the loss function of the YOLOv5 network model is set to obtain the improved YOLOv5 network model.

[0019] Furthermore, the context guidance block module is used to generate joint features based on the local features of the extracted image and the context information around the receiving field, and to enhance the generated joint features through weighted processing.

[0020] The high-resolution feature pyramid network feature fusion structure is used to fuse joint features of different scales after enhancement processing to obtain the final attention features.

[0021] Furthermore, the context bootstrap block module includes:

[0022] Local feature extractor, used to learn local features of an image using standard convolutional layers;

[0023] A surrounding context extractor is used to learn the surrounding context of the receptive field using dilated / dilated convolutional layers;

[0024] A joint feature extractor is used to obtain joint features by fusing the output of local features and the surrounding context through connection layers, batch normalization, and parameterization.

[0025] The global context extractor aggregates global context using a global average pooling layer, extracts global context using a multilayer perceptron, and performs weighted processing on the global context and joint features.

[0026] Furthermore, the fusion of enhanced joint features at different scales yields the final attention features, specifically:

[0027] Based on the enhanced processing results, a high-level feature and a low-scale feature are given respectively;

[0028] High-level features are expanded using transposed convolution;

[0029] The dimensions of the expanded high-scale features and low-scale features are unified, and bilinear interpolation is used to sample the expanded high-scale features.

[0030] Based on the sampling results, high-level features are converted into corresponding attention weights to filter out low-scale features;

[0031] The filtered low-scale features and the expanded high-scale features are fused to obtain the final attention features.

[0032] Furthermore, the final expression for the attention feature is as follows:

[0033] f att =BL(T-Conν(f) kigh ))

[0034] f a ' tt =flow*CA(fart)+fatt

[0035] Among them, f a ' tt f represents the final attention feature. att The initial attention features are represented by BL, BlockLayer, T, Conν(), and f. kigh f represents high-scale features low Represents low-scale features, CA represents the attention module, f art This represents the final attentional features after processing by the attention mechanism.

[0036] Furthermore, the expression for the loss function of the improved YOLOv5 network model is as follows:

[0037]

[0038] Where SHAPE-IOU represents the loss function of the improved YOLOv5 network model, IoU represents the intersection-union ratio between the predicted bounding box and the ground truth bounding box, and ρ(b,b) represents the intersection-union ratio between the predicted bounding box and the ground truth bounding box. g ) represents the Euclidean distance between the centers of the predicted bounding box and the ground truth bounding box, ρ represents the shape similarity between the predicted and ground truth bounding boxes, and b represents the width and height of the predicted bounding box. gdenoted by , where represents the width and height of the true bounding box, c represents the diagonal length of the smallest closure containing both the predicted and true bounding boxes, α represents the weighting coefficient, and v represents the shape cost term.

[0039] The beneficial effects of this invention are:

[0040] This invention provides a fruit defect detection method based on an improved YOLOv5 model. It enhances the data through Mosiac and Cutout algorithms, improves the convolutional model by replacing it with a CG block (Context Guided Block) module to enhance the feature extraction process, and improves the Neck FPN (Feature Pyramid Network) structure by replacing it with an HS-FPN (High-Resolution Feature Pyramid Network) structure to increase feature fusion. The loss function ShapeLou is optimized to accelerate model convergence and improve performance. The improved YOLO model is trained using the input dataset to obtain the improved model. Finally, the model is used to detect defects in images and videos of the target region. Testing shows that the real-time performance optimization of this invention enables the model to respond responsively, adjust model parameters according to actual needs, and is suitable for various security detection scenarios, enabling more accurate detection of fruit defects. Attached Figure Description

[0041] Figure 1 This is a flowchart of the method of the present invention.

[0042] Figure 2 This is an overview diagram of the Context Guided block module used in this invention.

[0043] Figure 3 This is a diagram of the YOLOv5 network structure used in this invention.

[0044] Figure 4 The curve shows a comparison between the original YOLOv5 model and the mAP@0.5 value of this invention.

[0045] Figure 5 This is a screenshot of the detection results from the original YOLOv5 model.

[0046] Figure 6 This is a diagram showing the detection effect of the present invention. Detailed Implementation

[0047] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0048] Example

[0049] like Figure 1 As shown, the present invention provides a method for detecting fruit defects, the implementation method of which is as follows:

[0050] S1. Collect image data containing fruit defect detection under different conditions, clean the data, generate an initial dataset, and label it;

[0051] In this embodiment, the collected image data is uniformly set to 640×640×3, and the data is labeled using the labelimg tool, and the label box information is saved.

[0052] S2. The labeled image data is preprocessed and divided into training, validation, and test sets. The implementation method is as follows:

[0053] S201. Randomly select an image and stitch together the coordinates of the reference points;

[0054] S202. Randomly select four images from the labeled image data, and based on the reference point, adjust the size and position of the four images and place them at four positions in a layer of a specified size, and map the relationship to the image labels.

[0055] S203. By stitching together the images in the layers, the coordinates of the out-of-bounds detection box are cropped;

[0056] S204. Randomly crop a square region of a fixed size from the latitude and longitude of the image and fill the region with zero values ​​to complete the preprocessing of the image data.

[0057] S205. Divide the preprocessed image data into training set, validation set and test set.

[0058] In this embodiment, the labeled dataset is augmented using Mosiac and Cutout methods, while the images are randomly rotated, cropped, and scaled. The specific steps are as follows:

[0059] In the Mosiac algorithm: the coordinates of the image stitching reference point (xc, yc) are randomly selected, and four images are randomly selected from the dataset. Based on the reference point, after adjusting their size and position, they are placed in four positions in a layer of a specified size. The mapping relationship is mapped to the image label. Based on the coordinates, the large images in the layer are stitched together, and the coordinates of the out-of-bounds detection boxes are processed.

[0060] In the Cutout algorithm: a fixed-size square region is randomly cropped from the image, and then zero values ​​are filled into this region. By occluding part of the image information, the neural network is forced to focus on other parts of the image.

[0061] S3. Construct an improved YOLOv5 network model, as follows: Figure 3 As shown, it is specifically as follows:

[0062] The convolutional module is improved by using the context-guided block module, the feature pyramid network structure is improved into a high-resolution feature pyramid network feature fusion structure, and the loss function of the YOLOv5 network model is set to obtain the improved YOLOv5 network model.

[0063] The context guidance block module is used to generate joint features based on the local features of the extracted image and the context information around the receiving field, and to enhance the generated joint features through weighted processing.

[0064] A high-resolution feature pyramid network feature fusion structure is used to fuse joint features of different scales after enhancement processing to obtain the final attention features;

[0065] In this embodiment, the context bootstrap block module includes:

[0066] Local feature extractor, used to learn local features of an image using standard convolutional layers;

[0067] A surrounding context extractor is used to learn the surrounding context of the receptive field using dilated / dilated convolutional layers;

[0068] A joint feature extractor is used to obtain joint features by fusing the output of local features and the surrounding context through connection layers, batch normalization, and parameterization.

[0069] The global context extractor aggregates global context using a global average pooling layer, extracts global context using a multilayer perceptron, and performs weighted processing on the global context and joint features.

[0070] In this embodiment, the joint features of different scales after enhancement processing are fused to obtain the final attention features, which are as follows:

[0071] Based on the enhanced processing results, a high-level feature and a low-scale feature are given respectively;

[0072] High-level features are expanded using transposed convolution;

[0073] The dimensions of the expanded high-scale features and low-scale features are unified, and bilinear interpolation is used to sample the expanded high-scale features.

[0074] Based on the sampling results, high-level features are converted into corresponding attention weights to filter out low-scale features;

[0075] The filtered low-scale features and the expanded high-scale features are fused to obtain the final attention features.

[0076] In this embodiment, the convolution is improved. The Context Guided block (CG block) module is a module that replaces Conv. Its basic principle is to simulate the human visual system's reliance on contextual information to understand the scene. The context guided block module is used to capture local features, surrounding context, and global context, and fuse this information to improve accuracy.

[0077] In this embodiment, the improvement process of the context bootstrap block module is as follows:

[0078] Local Feature Extractor (FLOC): Learns local features using standard convolutional layers.

[0079] Surrounding Context Extractor (FSUR): Uses dilated / dilated convolutional layers to learn the surrounding context of a larger receptive field.

[0080] Joint Feature Extractor (FJOI): It obtains joint features by fusing the output of local features and surrounding context through connected layers, batch normalization (BN), and parameterized ReLU (PReLU) operations.

[0081] Global Context Extractor (FGLO): Aggregates global context using a global average pooling layer and further extracts global context through a multilayer perceptron. It uses a scaling layer to weight the joint features with the extracted global context to emphasize useful components and suppress useless components.

[0082] In this embodiment, the Neck structure in YOLOv5 is improved to a High-Resolution Feature Pyramid Network (HS-FPN) structure. Selective Feature Fusion (SSF) is a key component of the HS-FPN, and its main function is to fuse feature maps of different scales. The specific implementation steps are as follows:

[0083] Given an input high-level feature, it is represented as:

[0084] Given a low-scale feature, represented as:

[0085] High-level features are first expanded using a transposed convolution of size 2 with a kernel size of 3x3, denoted as: f high / epsilonR C*2H*2W .

[0086] To unify the dimensions of high-level and low-scale features, bilinear interpolation is used to upsample or downsample high-level features, as follows:

[0087] The CA module is used to convert high-level features into corresponding attention weights to filter low-scale features; the filtered low-scale features are then fused with the high-level features to enhance the model's feature representation and obtain... The equations illustrate the feature selection fusion process:

[0088] f att =BL(T-Conν(f) kigh ))

[0089] f a ' tt =flow*CA(fart)+fatt

[0090] Among them, f a ' tt f represents the final attention feature. att The initial attention features are represented by BL, BlockLayer, T, Conν(), and f. kigh f represents high-scale features low Represents low-scale features, CA represents the attention module, f art This represents the final attention feature after processing by the attention mechanism. f represents the feature, high represents high scale, low represents low scale, epsilon represents the degree of model parameter adjustment, C, H, and W represent the number of channels, height, and width of the image, respectively, att represents the attention mechanism, and BL represents the Block Layer, which is a module composed of multiple layers.

[0091] In this embodiment, the context-guided block module and the high-resolution feature pyramid network feature fusion structure work together to improve the detection accuracy of fruit defects. The context-guided block module focuses on extracting local features from the image and combining them with surrounding contextual information to generate joint features. These features are then strengthened through weighted processing, enabling them to more accurately reflect subtle changes and defects on the fruit surface. This weighted processing not only enhances the expressiveness of local features but also incorporates information from the global context, allowing the model to understand the image content more comprehensively. Meanwhile, the high-resolution feature pyramid network feature fusion structure plays a crucial role in this process. By fusing feature maps of different scales, it allows the model to simultaneously focus on the large-scale structure and detailed features of fruit defects. This fusion strategy not only improves the resolution of features but also enhances the model's ability to identify defects of different sizes, especially when dealing with small and dense targets. Combining the advantages of these two structures, the improved YOLOv5 network model can more effectively extract key information about fruit defects from images and accurately identify these defects in the final detection results. This improvement not only enhances detection speed and accuracy but also enables the model to adapt to more complex real-world application scenarios, such as accurate fruit defect detection under varying lighting conditions and background interference. Specifically, the context guidance block module and the high-resolution feature pyramid network (HS-FPN) feature fusion structure work collaboratively within the network model, exhibiting a complementary and enhancing relationship. Within these two structures, the context guidance block module provides rich contextual information that can be utilized by the HS-FPN feature fusion structure to better understand and fuse features at different scales. The weighted processing result (i.e., the output of the context guidance block module) can serve as one of the inputs to the HS-FPN to enhance the feature fusion process.

[0092] In this embodiment, the shape and scale of the bounding box itself will affect the regression results. By using the Shape Lou method, the loss can be calculated by focusing on the shape and scale of the bounding box itself, thus making the bounding box regression more accurate.

[0093] S4. Train and validate the improved YOLOv5 network model using the training and validation sets;

[0094] In this embodiment, after the final round of model training, a weight file and an "average precision" table (such as...) are obtained. Figure 4 As shown in the figure, the generated best weight file is passed into the inference function for improvement and adjustment.

[0095] S5. Based on the test set, use the trained and validated YOLOv5 network model to detect fruit defect images and obtain the detection results (e.g., Figure 5 , Figure 6 (As shown).

[0096] In this embodiment, based on the test set, image data containing fruit defects is collected and preprocessed, and an improved YOLOv5 network model is used. This model integrates the context guide block (CG block) and the high-resolution feature pyramid network (HS-FPN) to enhance feature extraction and fusion, and accelerates the convergence of model training by optimizing the loss function, ultimately achieving efficient detection of fruit defects such as scars and damage.

[0097] In this embodiment, the test dataset is fed into the model for testing, and the detection result image is obtained.

[0098] In this embodiment, as Figure 2 As shown, this embodiment uses YOLOv5 as the base model and improves its backbone network by replacing the C3 structure in the original network with the C3_ContextDown structure. This can effectively aggregate context information from the bottom to the top layer and capture context information at both the semantic level (from the deep layer) and the spatial level (from the shallow layer). The width_multiple (width multiplier) is changed to 0.25, which reduces the amount of computation and improves the inference speed while maintaining high accuracy.

[0099] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting fruit defects, characterized in that, Includes the following steps: S1. Collect image data containing fruit defect detection under different conditions, clean the data, generate an initial dataset, and label it; S2. Preprocess the labeled image data and divide it into training set, validation set and test set; S3. Construct an improved YOLOv5 network model; S4. Train and validate the improved YOLOv5 network model using the training and validation sets; S5. Based on the test set, use the trained and validated YOLOv5 network model to detect fruit defect images and obtain the detection results.

2. The fruit defect detection method according to claim 1, characterized in that, S2 includes the following steps: S201. Randomly select an image and stitch together the coordinates of the reference points; S202. Randomly select four images from the labeled image data, and based on the reference point, adjust the size and position of the four images and place them at four positions in a layer of a specified size, and map the relationship to the image labels. S203. By stitching together the images in the layers, the coordinates of the out-of-bounds detection box are cropped; S204. Randomly crop a square region of a fixed size from the latitude and longitude of the image and fill the region with zero values ​​to complete the preprocessing of the image data. S205. Divide the preprocessed image data into training set, validation set and test set.

3. The fruit defect detection method according to claim 1, characterized in that, Specifically, S3 is: The convolutional module is improved by using the context-guided block module, the feature pyramid network structure is improved into a high-resolution feature pyramid network feature fusion structure, and the loss function of the YOLOv5 network model is set to obtain the improved YOLOv5 network model.

4. The fruit defect detection method according to claim 3, characterized in that, The context guidance block module is used to generate joint features based on the local features of the extracted image and the context information around the receiving field, and to enhance the generated joint features through weighted processing. The high-resolution feature pyramid network feature fusion structure is used to fuse joint features of different scales after enhancement processing to obtain the final attention features.

5. The fruit defect detection method according to claim 4, characterized in that, The context bootstrap block module includes: Local feature extractor, used to learn local features of an image using standard convolutional layers; A surrounding context extractor is used to learn the surrounding context of the receptive field using dilated / dilated convolutional layers; A joint feature extractor is used to obtain joint features by fusing the output of local features and the surrounding context through connection layers, batch normalization, and parameterization. The global context extractor aggregates global context using a global average pooling layer, extracts global context using a multilayer perceptron, and performs weighted processing on the global context and joint features.

6. The fruit defect detection method according to claim 4, characterized in that, The fusion of enhanced joint features at different scales yields the final attention features, specifically: Based on the enhanced processing results, a high-level feature and a low-scale feature are given respectively; High-level features are expanded using transposed convolution; The dimensions of the expanded high-scale features and low-scale features are unified, and bilinear interpolation is used to sample the expanded high-scale features. Based on the sampling results, high-level features are converted into corresponding attention weights to filter out low-scale features; The filtered low-scale features and the expanded high-scale features are fused to obtain the final attention features.

7. The fruit defect detection method according to claim 6, characterized in that, The expression for the final attention feature is as follows: f att =BL(T-Conν(f kigh )) in a ' tt =flow*CA(fart)+fact Among them, f a ' tt f represents the final attention feature. att The initial attention features are represented by BL, BlockLayer, T, Conν(), and f. kigh f represents high-scale features low Represents low-scale features, CA represents the attention module, f art This represents the final attentional features after processing by the attention mechanism.

8. The fruit defect detection method according to claim 3, characterized in that, The expression for the loss function of the improved YOLOv5 network model is as follows: Where SHAPE-IOU represents the loss function of the improved YOLOv5 network model, IoU represents the intersection-union ratio between the predicted bounding box and the ground truth bounding box, and ρ(b,b) represents the intersection-union ratio between the predicted bounding box and the ground truth bounding box. g ) represents the Euclidean distance between the centers of the predicted bounding box and the ground truth bounding box, ρ represents the shape similarity between the predicted and ground truth bounding boxes, and b represents the width and height of the predicted bounding box. g denoted by , where represents the width and height of the true bounding box, c represents the diagonal length of the smallest closure containing both the predicted and true bounding boxes, α represents the weighting coefficient, and v represents the shape cost term.

Images