Navel orange crack detection method based on YOLOV7 neural network with attention mechanism

Abstract

The present invention relates to a navel orange fruit crack detection method using a YOLOv7 neural network with an attention mechanism, comprising the following steps: S1, collecting navel orange fruit crack images, labeling the cracked fruit, forming a navel orange fruit crack dataset, and preprocessing the dataset; S2, improving the YOLOv7 neural network by adding a convolutional attention mechanism module; S3, inputting the labeled dataset into the improved YOLOv7 neural network in a format that meets network requirements for training and testing the results; and S4, deploying the trained model to a mobile terminal for navel orange fruit crack detection and identification. The navel orange fruit crack detection network model constructed by the present invention is based on the YOLOv7 network structure and embeds a convolutional attention mechanism, thereby achieving higher accuracy and faster detection speed in fruit crack identification, effectively resolving the current problem of misdetection of fruit cracks, navel stems, and navel bellies in navel orange fruit crack detection.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence detection, and more specifically, to a navel orange fruit cracking detection method using a YOLOV7 neural network with an attention mechanism. Background Art

[0002] China is a major producer and consumer of navel oranges. Currently, post-harvest grading of navel oranges still primarily relies on manual screening, which is labor-intensive and time-consuming. Cracked navel oranges are the most common defective fruit in navel orange sorting, as they are highly perishable during transportation, storage, and distribution. Current machine vision inspection solutions are prone to misdetecting cracked navels and misidentified navels and stems. Traditional machine vision inspection models are also slow. YOLOv7, a next-generation object detection model, offers advantages such as fast detection speed and high accuracy. To address these shortcomings of existing machine vision inspection solutions, we propose a cracked navel orange detection method based on an improved YOLOv7 neural network. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a navel orange cracked fruit detection method using a YOLOV7 neural network with an attention mechanism, which can improve both detection accuracy and detection speed.

[0004] The technical solution adopted by the present invention to solve the technical problem is to construct a navel orange cracking detection method using a YOLOV7 neural network with an attention mechanism, comprising the following steps:

[0005] S1. Collect navel orange cracked fruit images and mark cracked fruit labels to form a navel orange cracked fruit dataset, and preprocess the dataset;

[0006] S2. Improve the YOLOv7 neural network and add a convolutional attention mechanism module;

[0007] S3, inputting the labeled data set into the improved YOLOv7 neural network in a format that meets the network requirements for training and testing the results;

[0008] S4. Deploy the trained model to the mobile terminal to detect and identify cracked navel oranges.

[0009] According to the above scheme, in step S1, the method for collecting images of navel orange cracked fruit includes:

[0010] The cracked navel orange fruit was placed horizontally upward, and images of the cracked navel orange fruit were taken horizontally, vertically, and at a 45-degree angle at the cracked navel orange fruit; the resolution of the captured images was 1440x1080 pixels.

[0011] According to the above scheme, in step S1, the preprocessing of the data set includes:

[0012] Annotate the image dataset using the image annotation tool Labelimg;

[0013] The annotated image dataset was saved as a txt file using the YOLO official dataset format, and the dataset was divided into a training set and a test set according to a ratio of 8:2.

[0014] According to the above scheme, in step S2, the convolutional attention mechanism module is added including a channel attention module and a spatial attention module.

[0015] According to the above scheme, the channel attention module includes:

[0016] The input feature map, i.e. H×W×C, is subjected to the global maximum pooling layer and global average pooling based on H and W respectively to obtain two 1×1×C feature maps; the feature maps are respectively sent to a two-layer neural network, the number of neurons in the first layer is C / r, the activation function is ReLU, and the number of neurons in the second layer is C; the output features are element-wise summed and activated to generate a channel attention module, i.e. M c ; M c Multiply the input features to generate the input features required by the spatial attention module. The specific implementation formula is as follows:

[0017]

[0018] Among them, H and W are the height and width of the input feature map respectively. Represents the Sigmoid function, F represents the input feature, and Respectively represent the features after the average pooling and the maximum pooling operations, and Represent the weight function of each layer of the neural network.

[0019] According to the above scheme, the spatial attention module includes:

[0020] The features output by the channel attention module are used as the input feature map of the spatial attention module; the input feature map passes through the maximum pooling layer and the average pooling layer to obtain two H*W*1 feature maps; the two feature maps are concat fused and passed through a 7*7 convolution layer to reduce the dimension to one channel, and then pass through the Sigmoid function to generate the spatial attention module, i.e., M s ; M s Multiply the input feature map of the spatial attention module to obtain the final generated features. The specific implementation formula is as follows:

[0021]

[0022] in, represents the Sigmoid function, Represents a convolution operation of size 7*7, F represents the input feature, and Represent the features after average pooling and maximum pooling operations respectively.

[0023] According to the above scheme, in step S3, the process of training the YOLOv7 neural network and testing the results includes:

[0024] Mosaic data enhancement is used at the input end to splice the input images by randomly scaling, randomly cropping, and randomly arranging them;

[0025] Adaptive anchor box calculation;

[0026] Adaptive image scaling.

[0027] According to the above scheme, in step S3, the improved YOLOv7 neural network structure includes:

[0028] Input module: Input 3*640*640 three-channel color image;

[0029] Backbone module: used for feature extraction, composed of several CBS layers, E-ELAN layers, MPConv layers and CBAM layers, among which the CBS layer consists of convolution layer + BN layer + activation function, and uses ReakyRelu as the activation function; E-ELAN is an efficient layer aggregation network that can improve the learning ability of the network without destroying the original gradient path; the MPConv convolution layer adds a maximum pooling layer on the basis of the BCov layer to form two upper and lower branches, and finally uses the Concat operation to fuse the features extracted from the upper and lower branches to improve the network feature extraction ability; the CBAM layer includes a channel attention module and a spatial attention module. The channel attention module uses maximum pooling and average pooling to compress the feature map to obtain two different spatial background descriptions, which are calculated using a shared network composed of MLP; the spatial attention module uses maximum pooling and average pooling in the channel dimension to obtain two different feature descriptions, uses concat for feature fusion, and uses convolution operation to generate a spatial attention module;

[0030] Head module: used for prediction. First, the SPP pyramid structure is used to make the head network suitable for multi-scale input. Then, the aggregated feature pyramid network structure is adopted to pass the bottom-level information to the upper layer along the bottom-up path to achieve the fusion of features at different levels. Finally, the REPcon structure is used to adjust the number of channels of features at different scales.

[0031] According to the above scheme, in step S4, the trained weight model is deployed to the mobile terminal, the video is obtained through the camera and input into the mobile terminal, the navel orange cracked fruits appearing in the video are detected in real time, and the detection results are output; according to the detection results, the cracked navel orange fruits are picked out in conjunction with the robotic arm.

[0032] The implementation of the YOLOV7 neural network navel orange crack detection method with the addition of an attention mechanism of the present invention has the following beneficial effects:

[0033] The navel orange crack detection network model constructed in the present invention is based on the YOLOv7 network structure and is embedded with a convolutional attention mechanism, thereby making the network more accurate and faster in fruit crack identification, and effectively solving the current problem of easy misdetection of cracked fruit, navel stem, and navel belly in navel orange crack detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0035] Figure 1 A schematic diagram of the basic process of a method for detecting navel orange fruit cracking using a YOLOv7 neural network with an attention mechanism provided by one embodiment of the present invention;

[0036] Figure 2 A schematic diagram illustrating the overall implementation of a convolutional attention module for a method for detecting navel orange fruit cracking using a YOLOv7 neural network with an attention mechanism, provided in accordance with one embodiment of the present invention;

[0037] Figure 3 A schematic diagram illustrating an implementation of a channel attention module for a method for detecting navel orange cracked fruit using a YOLOv7 neural network with an attention mechanism, provided in accordance with one embodiment of the present invention;

[0038] Figure 4 A schematic diagram illustrating an implementation of a spatial attention module for a method for detecting cracked navel oranges using a YOLOv7 neural network with an attention mechanism, according to one embodiment of the present invention;

[0039] Figure 5 A schematic diagram of the YOLOv7 structure of a method for detecting navel orange fruit cracking using a YOLOv7 neural network with an attention mechanism provided by one embodiment of the present invention;

[0040] Figure 6A schematic diagram of detection results after improved network training of a method for detecting navel orange fruit cracking using a YOLOv7 neural network with an attention mechanism provided by one embodiment of the present invention;

[0041] Figure 7 A schematic diagram of detection results after improved network training of a navel orange fruit cracking detection method using a YOLOv7 neural network with an attention mechanism provided in another embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, specific embodiments of the present invention are now described in detail with reference to the accompanying drawings.

[0043] Example 1

[0044] Reference Figures 1 to 6 , as one embodiment of the present invention, provides a method for detecting navel orange cracked fruit using a YOLOv7 neural network with an attention mechanism, comprising the following steps:

[0045] S1: Collect navel orange cracked fruit images, annotate cracked fruit labels, form a navel orange cracked fruit dataset, and preprocess the dataset;

[0046] It should be noted that the image acquisition method includes: placing the navel orange cracking position horizontally upward, and photographing the navel orange cracking image in the horizontal direction, vertical direction and 45-degree angle direction respectively; the photographed image resolution is 1440x1080 pixels; the navel orange variety is red-heart navel orange, also known as Cara Cara red-fleshed navel orange; it should be noted that the dataset annotation and preprocessing method includes: using the image annotation tool Labelimg to annotate the image dataset; using the YOLO official specified dataset format to save the annotated image dataset as a txt format annotation file, and dividing the dataset into a training set and a test set according to a ratio of 8:2.

[0047] S2: Improve the YOLOv7 neural network and add a convolutional attention mechanism module;

[0048] It should be noted that, for example, adding a channel attention module includes: passing the input feature map, i.e., H×W×C, through the global maximum pooling layer and global average pooling based on H and W, respectively, to obtain two 1×1×C feature maps; the feature maps are respectively sent to a two-layer neural network, the number of neurons in the first layer is C / r, the activation function is ReLU, and the number of neurons in the second layer is C; the output features are element-wise summed and activated to generate a channel attention module, i.e., M c ; M c Multiply the input features to generate the input features required by the spatial attention module. The specific implementation formula is as follows:

[0049]

[0050]

[0051] Among them, H and W are the height and width of the input feature map respectively, σ represents the Sigmoid function, and F represents the input feature. and Respectively represent the features after the average pooling and the maximum pooling operations, and Represent the weight function of each layer of the neural network.

[0052] Adding a spatial attention module includes taking the features output by the channel attention module as the input feature map of the spatial attention module; the input feature map passes through the maximum pooling layer and the average pooling layer to obtain two H*W*1 feature maps; the two feature maps are concat-fused and passed through a 7*7 convolution layer to reduce the dimension to one channel, and then pass through the Sigmoid function to generate the spatial attention module, i.e., M. s ; M s Multiply the input feature map of the spatial attention module to obtain the final generated features. The specific implementation formula is as follows:

[0053]

[0054]

[0055] in, represents the Sigmoid function, Represents a convolution operation of size 7*7, F represents the input feature, and Represent the features after average pooling and maximum pooling operations respectively.

[0056] S3: Input the labeled data set into the improved YOLOv7 neural network in a format that meets the network requirements for training and testing the results.

[0057] It should be noted that Mosaic data enhancement is used at the input end to perform random scaling, random cutting, and random arrangement on multiple input images for splicing; adaptive anchor frame calculation, in the YOLO model, there will be an initial set length and width anchor frame for different data sets. During network training, the network will output a predicted frame based on the initial anchor frame, and then compare it with the real frame Groundtruth, calculate the gap between the two, and then update it in reverse, and continuously iterate the network parameters. In previous YOLO versions, it was necessary to run this anchor frame program separately to achieve the effect. In YOLOv7, the adaptive anchor frame is used. The anchor box program is embedded in the overall training program, which speeds up data processing and training. The optimal anchor box in the training set can be adaptively calculated during each training session. Adaptive image scaling: In commonly used detection algorithms, the input original image is uniformly scaled to a standard size before being fed into the detection network. However, due to the different aspect ratios of many images, images that do not reach a resolution of 640*640 are scaled and padded, resulting in some images with more black edges and information redundancy, which affects the inference speed. YOLOv7 has modified the letterbox function to adaptively add minimal black edges to the original image, improving the speed of target detection.

[0058] It should be noted that the detailed explanation of each module of YOLOv7 is as follows:

[0059] Input module: Input a 3*640*640 three-channel color image.

[0060] Backbone module: used for feature extraction, composed of several CBS layers, E-ELAN layers, MPConv layers and CBAM layers, among which the CBS layer consists of convolution layer + BN layer + activation function, and uses ReakyRelu as the activation function; E-ELAN is an efficient layer aggregation network that can improve the learning ability of the network without destroying the original gradient path; the MPConv convolution layer adds a maximum pooling layer on the basis of the BCov layer to form two upper and lower branches, and finally uses the Concat operation to fuse the features extracted from the upper and lower branches to improve the network feature extraction ability; the CBAM layer includes a channel attention module and a spatial attention module. The channel attention module uses maximum pooling and average pooling to compress the feature map to obtain two different spatial background descriptions, and uses a shared network composed of MLP to calculate the two different spatial background descriptions; the spatial attention module uses maximum pooling and average pooling in the channel dimension to obtain two different feature descriptions, uses concat for feature fusion, and uses convolution operation to generate the spatial attention module.

[0061] Head module: used for prediction. First, the SPP pyramid structure is used to make the head network suitable for multi-scale input. Then, the aggregated feature pyramid network structure is adopted to pass the bottom-level information to the upper layer along the bottom-up path to achieve the fusion of features at different levels. Finally, the REPcon structure is used to adjust the number of channels of features at different scales.

[0062] S4: Deploy the trained model to the mobile terminal to detect and identify cracked navel oranges.

[0063] Deploy the trained weight model to the mobile terminal, capture the video through the camera and input it into the mobile terminal, perform real-time detection of navel orange cracks in the video, and output the detection results;

[0064] S5: Based on the test results, the robot arm is used to pick out the cracked navel oranges.

[0065] Example 2

[0066] This embodiment is another embodiment of the present invention. Unlike the first embodiment, this embodiment provides a verification test of a method for identifying cracked navel orange fruit using a YOLOv7 network with an attention mechanism. To verify and illustrate the technical effects of this method, this embodiment conducts comparative tests between an existing solution and the solution of the present invention, and compares the test results using scientific demonstration to verify the actual effects of this method.

[0067] This example uses an Intel(R) Xeon(R) Gold 5117 CPU @ 2.00GHz, an NVIDIA Tesla PH402 SKU 200 GPU for training, Pytorch 1.9.0 as the deep learning framework, and CUDA version 11.1. Following the steps of Example 1, the image data is input into the unmodified YOLOv7 network for training and testing. The results of the two tests are compared, and the results are shown in Table 1:

[0068] Table 1: Comparison of experimental results.

[0069]

[0070] Among them, mAP refers to the average of the average precision values ​​of all categories, which can reflect the accuracy of the model. FPS is the number of image frames that the network model can recognize per second. It can be seen that the recognition accuracy of the model with the addition of the attention mechanism has been improved, and the recognition rate of the model has also been improved. In addition, the convergence speed of the model during training has also been significantly improved.

[0071] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are protected by the present invention.

Claims

1. A method for detecting navel orange fruit cracking using a YOLOV7 neural network with an attention mechanism, characterized in that: The following steps are involved: S1. Collect navel orange cracked fruit images and mark cracked fruit labels to form a navel orange cracked fruit dataset, and preprocess the dataset; S2. Improve the YOLOv7 neural network and add a convolutional attention mechanism module; S3. Input the labeled data set into the improved YOLOv7 neural network in a format that meets the network requirements for training and testing the results; In step S3, the improved YOLOv7 neural network structure includes: Input module: Input 3*640*640 three-channel color image; Backbone module: used for feature extraction, composed of several CBS layers, E-ELAN layers, MPConv layers and CBAM layers, among which the CBS layer consists of convolution layer + BN layer + activation function, and uses ReakyRelu as the activation function; E-ELAN is an efficient layer aggregation network that can improve the learning ability of the network without destroying the original gradient path; the MPConv convolution layer adds a maximum pooling layer on the basis of the BCov layer to form two upper and lower branches, and finally uses the Concat operation to fuse the features extracted from the upper and lower branches to improve the network feature extraction ability; the CBAM layer includes a channel attention module and a spatial attention module. The channel attention module uses maximum pooling and average pooling to compress the feature map to obtain two different spatial background descriptions, which are calculated using a shared network composed of MLP; the spatial attention module uses maximum pooling and average pooling in the channel dimension to obtain two different feature descriptions, uses concat for feature fusion, and uses convolution operation to generate a spatial attention module; Head module: used for prediction. First, the SPP pyramid structure is used to make the head network suitable for multi-scale input. Then, the aggregated feature pyramid network structure is adopted to transfer the bottom-level information to the upper layer along the bottom-up path, realizing the fusion of features at different levels. Finally, the REPcon structure is used to adjust the number of channels of features at different scales. S4. Deploy the trained model to the mobile terminal to detect and identify cracked navel oranges.

2. The method for detecting navel orange fruit cracking using a YOLOV7 neural network with an attention mechanism according to claim 1, wherein: In step S1, the method for collecting images of navel orange cracked fruit includes: The cracked navel orange fruit was placed horizontally upward, and images of the cracked navel orange fruit were taken horizontally, vertically, and at a 45-degree angle at the cracked navel orange fruit; the resolution of the captured images was 1440x1080 pixels.

3. The method for detecting navel orange fruit cracking using a YOLOV7 neural network with an attention mechanism according to claim 1, wherein: In step S1, the preprocessing of the data set includes: Use the image annotation tool Labelimg to annotate the dataset; The labeled dataset was saved as a txt file using the YOLO official dataset format, and the dataset was divided into a training set and a test set according to a ratio of 8:

2.

4. The method for detecting navel orange fruit cracking using a YOLOV7 neural network with an attention mechanism according to claim 1, wherein: In step S2, the convolutional attention mechanism module is added including a channel attention module and a spatial attention module.

5. The method for detecting navel orange fruit cracking using a YOLOV7 neural network with an attention mechanism according to claim 4, wherein: The channel attention module includes: The input feature map, i.e. H×W×C, is subjected to the global maximum pooling layer and global average pooling based on H and W respectively to obtain two 1×1×C feature maps; the feature maps are respectively sent to a two-layer neural network, the number of neurons in the first layer is C / r, the activation function is ReLU, and the number of neurons in the second layer is C; the output features are element-wise summed and activated to generate a channel attention module, i.e. M c ; M c Multiply the input features to generate the input features required by the spatial attention module. The specific implementation formula is as follows: Among them, H and W are the height and width of the input feature map respectively. Represents the Sigmoid function, F represents the input feature, and Respectively represent the features after the average pooling and the maximum pooling operations, and Represent the weight function of each layer of the neural network.

6. The method for detecting navel orange fruit cracking using a YOLOV7 neural network with an attention mechanism according to claim 5, wherein: The spatial attention module includes: The features output by the channel attention module are used as the input feature map of the spatial attention module; the input feature map passes through the maximum pooling layer and the average pooling layer to obtain two H*W*1 feature maps; the two feature maps are concat fused and passed through a 7*7 convolution layer to reduce the dimension to one channel, and then pass through the Sigmoid function to generate the spatial attention module, i.e., M s ; M s Multiply the input feature map of the spatial attention module to obtain the final generated features. The specific implementation formula is as follows: in, represents the Sigmoid function, Represents a convolution operation of size 7*7, F represents the input feature, and Represent the features after average pooling and maximum pooling operations respectively.

7. The method for detecting navel orange fruit cracking using a YOLOV7 neural network with an attention mechanism according to claim 1, wherein: In step S3, the process of training the YOLOv7 neural network and testing the results includes: Mosaic data enhancement is used at the input end to splice the input images by randomly scaling, randomly cropping, and randomly arranging them; Adaptive anchor box calculation; Adaptive image scaling.

8. The method for detecting navel orange fruit cracking using a YOLOV7 neural network with an attention mechanism according to claim 1, wherein: In step S4, the trained weight model is deployed to the mobile terminal, a video is acquired through a camera and input into the mobile terminal, the cracked navel orange fruits appearing in the video are detected in real time, and the detection results are output; based on the detection results, the cracked navel orange fruits are picked out in cooperation with a robotic arm.

Images