Improved Faster RCNN-based method and system for detecting maturity of myrica punctata

Abstract

The invention provides an improved Faster RCNN-based method for detecting maturity of myrica rubra, which comprises the following steps: acquiring and marking an image set of myrica rubra, and further dividing a training set, a verification set and a test set; on the basis of an improved Faster RCNN network model, constructing a maturity detection model of the myrica punctata; the improved Faster RCNN network model comprises a feature extraction backbone network, an RPN network, an RoI pooling layer and a full connection layer, wherein the feature extraction backbone network is composed of a ConvNeXt-T module and an FPN module, and an SE attention mechanism is introduced into the ConvNeXt-T module and the FPN module. Based on the training set, the verification set and the test set, training, verifying and testing the maturity detection model to obtain a trained maturity detection model; and obtaining a to-be-detected image of the symbiotic myrica rubra and importing the to-be-detected image into the trained maturity detection model to obtain that the maturity of the symbiotic myrica rubra corresponding to the to-be-detected image is one of immature, semi-mature, near-mature and full-mature. According to the invention, not only can the detection efficiency of the myrica dentata be improved, but also the detection precision can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer image technology, and in particular to a method and system for detecting the maturity of Dingao waxberry based on an improved Faster RCNN. Background Art

[0002] Dingao bayberry is a specialty of Ouhai District, Wenzhou City, Zhejiang Province. It is also a national geographical indication product of China. It enjoys the reputation of "red plate and green stem". It has high nutritional and medicinal value. However, the picking time of Dingao bayberry is very limited. If it is picked too early, the fruit will be small, of low value and poor flavor; if it is picked too late, the fruit will easily rot or fall off, resulting in reduced production and affecting the economic value of the commodity. Therefore, maturity becomes a crucial reference standard in the harvesting process of Dingao bayberry.

[0003] In recent years, in order to ensure the accuracy and reliability of fruit maturity detection, domestic and foreign scholars have proposed detection methods based on machine vision. In the detection of bayberry maturity, detection is mainly based on threshold segmentation, color space transformation, or combined with machine learning technology. For example, Xu Liming et al. used the SUSAN operator and Hough transform to realize the recognition of mature bayberry in natural scenes, used the maximum inter-class variance threshold segmentation method Otsu to segment the image, and then used the SUSAN operator feature for edge detection, and then used the Hough transform to realize the detection of bayberry maturity. For another example, Lei Huan et al. proposed a machine vision-based automatic detection method for mature bayberry to solve the problem of low recognition rate caused by the influence of lighting, occlusion, etc., segmented the bayberry area through adaptive histogram equalization, color difference method and OneCut algorithm, and used watershed transform and convex hull theory to separate overlapping fruits. This method can accurately detect mature bayberry fruits in orchard environments. For example, Huang Kai et al. proposed a machine vision algorithm based on multi-feature fusion, which extracted the texture features and color features of bayberry images and fused them in series, and then constructed a support vector machine (SVM) model to achieve efficient estimation of different maturity of bayberry. However, the above traditional machine vision detection methods rely heavily on feature engineering, usually require manual extraction of image features, and the algorithms often lack the ability to generalize to diverse and complex environments, thus affecting the stability of fruit maturity detection and recognition.

[0004] With the continuous improvement of deep learning theory and the continuous improvement of computer hardware performance, artificial intelligence technology has developed rapidly. Many image classification and recognition methods based on deep learning have been widely used in the agricultural field, especially in the direction of fruit maturity detection. For example, Zhu Xu et al. proposed a blueberry fruit maturity recognition method based on the Faster R-CNN model, which has a high recognition rate for ripe, semi-ripe and immature blueberries, and the recognition speed can also meet the requirements of real-time recognition. For another example, Wang Jing et al. introduced the YOLO algorithm into the recognition of jujube fruit maturity in a natural environment, and constructed four recognition models, namely YOLO V3, YOLO V4, YOLO V4-Tiny and Mobilenet-YOLO V4-Lite, which can better recognize the maturity of jujube fruit. For another example, Li P et al. used the MHSA attention mechanism to improve the backbone network of YOLOv8, and constructed a tomato fruit maturity grading model based on MHSA-YOLOv8, which can detect both online and offline, effectively improving the harvesting and grading efficiency of tomato growers. For example, Xiong Juntao et al. proposed a papaya maturity detection method based on the lightweight YOLO v5-Lite model. Compared with the mainstream lightweight detection algorithm, the mAP has been effectively improved. Experiments have shown that the model can better cope with papaya maturity detection under complex backgrounds. For example, Xiang Xinjian et al. proposed a bayberry maturity detection method based on the improved YOLOX-NANO algorithm. The average accuracy of the improved model reached 92.67%, and the model size increased by only 0.059 MB, which is more suitable for embedded device deployment.

[0005] Compared with traditional maturity detection methods, deep learning networks have shown more significant advantages in the field of maturity detection, and can achieve rapid and accurate detection of fruit maturity in complex natural growth environments. At present, there are few studies on Dingao Bayberry recognition based on deep learning, and they are mainly focused on the YOLO series of methods. Considering that Dingao Bayberry is planted densely, this leads to a variety of complex phenomena such as multiple fruit overlap, occlusion, and uneven lighting during the growth process, making the existing YOLO fruit recognition method unsatisfactory for Dingao Bayberry detection. Therefore, it is necessary to propose a new fruit recognition method to detect the maturity of Dingao Bayberry, which can not only improve the detection efficiency, but also improve the detection accuracy. Summary of the invention

[0006] The technical problem to be solved by the embodiments of the present invention is to provide a method and system for detecting the maturity of Dingao Bayberry based on the improved Faster RCNN, which can not only improve the detection efficiency of Dingao Bayberry, but also improve the detection accuracy.

[0007] In order to solve the above technical problems, an embodiment of the present invention provides a method for detecting the maturity of Dingao bayberry based on an improved Faster RCNN, the method comprising the following steps: Obtain a Dingao bayberry image set, and mark each image in the Dingao bayberry image set, and further divide it into a training set, a validation set, and a test set; wherein the Dingao bayberry image set is composed of unripe images of Dingao bayberry, semi-ripe images of Dingao bayberry, nearly ripe images of Dingao bayberry, and fully ripe images of Dingao bayberry; Based on the improved Faster RCNN network model, a maturity detection model for Dingao bayberry is constructed; wherein the improved Faster RCNN network model includes a feature extraction backbone network composed of a ConvNeXt-T module and an FPN module that introduces the SE attention mechanism, an RPN network, a RoI pooling layer, and a fully connected layer; the ConvNeXt-T module is used to extract key features of the input image to form a four-layer feature map; the FPN module is used to perform multi-scale feature fusion on the four-layer feature maps output by the ConvNeXt-T module to form a multi-scale feature atlas; Based on the training set, the maturity detection model is trained, and further based on the validation set and the test set, the maturity detection model is validated and tested to obtain a trained maturity detection model; The image to be tested of Dingao bayberry is obtained, and the image to be tested is imported into the trained maturity detection model to obtain the maturity of the Dingao bayberry corresponding to the image to be tested, which is one of immature, semi-mature, nearly mature and fully mature.

[0008] The ConvNeXt-T module includes a downsampling module and a four-stage backbone structure consisting of ConvNeXt Block modules; The downsampling module is located before the backbone structure, and is used to receive an input image, downsample the features of the input image, and further reduce the spatial size of the downsampled feature map by pooling or stride convolution, thereby increasing the receptive field and extraction capability of the features; The backbone structure is divided into four stages; each stage includes a ConvNeXt Block module, and each ConvNeXt Block module contains a 1x1 convolution layer and a 3x3 grouped convolution layer; the Block ratio of each stage is 1:1:3:1, and the corresponding number of Blocks is 3, 3, 9, and 3, and the number of channels is 96, 192, 384, and 768 respectively; each stage performs feature extraction through a 4x or 2x downsampling rate, and integrates the SE attention mechanism to form a corresponding feature map.

[0009] Wherein, each image in the Dingao waxberry image set and the image to be tested are subjected to image mosaic processing and image enhancement processing.

[0010] The image enhancement processing includes random image rotation, image translation, left-right flipping, image cropping and image scaling.

[0011] The embodiment of the present invention also provides a Dingao waxberry maturity detection system based on improved Faster RCNN, comprising: An image set construction unit is used to obtain a Dingao bayberry image set, mark each image in the Dingao bayberry image set, and further divide it into a training set, a validation set, and a test set; wherein the Dingao bayberry image set is composed of unripe images of Dingao bayberry, semi-ripe images of Dingao bayberry, nearly ripe images of Dingao bayberry, and fully ripe images of Dingao bayberry; A maturity detection model construction unit is used to construct a maturity detection model for Dingao bayberry based on an improved Faster RCNN network model; wherein the improved Faster RCNN network model includes a feature extraction backbone network, an RPN network, a RoI pooling layer, and a fully connected layer composed of a ConvNeXt-T module and an FPN module that introduces an SE attention mechanism; the ConvNeXt-T module is used to extract key features of an input image to form a four-layer feature map; the FPN module is used to perform multi-scale feature fusion on the four-layer feature maps output by the ConvNeXt-T module to form a multi-scale feature atlas; A maturity detection model training unit, used to train the maturity detection model based on the training set, and further verify and test the maturity detection model based on the verification set and the test set to obtain a trained maturity detection model; The maturity detection unit is used to obtain the image to be tested of Dingao bayberry, and import the image to be tested into the trained maturity detection model to obtain the maturity of the Dingao bayberry corresponding to the image to be tested as one of immature, semi-mature, nearly mature and fully mature.

[0012] The ConvNeXt-T module includes a downsampling module and a four-stage backbone structure consisting of ConvNeXt Block modules; The downsampling module is located before the backbone structure, and is used to receive an input image, downsample the features of the input image, and further reduce the spatial size of the downsampled feature map by pooling or stride convolution, thereby increasing the receptive field and extraction capability of the features; The backbone structure is divided into four stages; each stage includes a ConvNeXt Block module, and each ConvNeXt Block module contains a 1x1 convolution layer and a 3x3 grouped convolution layer; the Block ratio of each stage is 1:1:3:1, and the corresponding number of Blocks is 3, 3, 9, and 3, and the number of channels is 96, 192, 384, and 768 respectively; each stage performs feature extraction through a 4x or 2x downsampling rate, and integrates the SE attention mechanism to form a corresponding feature map.

[0013] Wherein, each image in the Dingao waxberry image set and the image to be tested are subjected to image mosaic processing and image enhancement processing.

[0014] The image enhancement processing includes random image rotation, image translation, left-right flipping, image cropping and image scaling.

[0015] Implementing the embodiments of the present invention has the following beneficial effects: The maturity detection model of the present invention is constructed based on an improved Faster RCNN network model. Since a ConvNeXt-T module and an FPN module of an SE attention mechanism are introduced into the improved Faster RCNN network model to form a feature extraction backbone network, the ConvNeXt-T module can be used to improve recognition performance in a complex environment, the SE attention mechanism is introduced to enhance the network's learning ability for different maturity characteristics of Dingao bayberry fruits, and the FPN module (feature pyramid network) is combined to improve the detection ability of small-sized Dingao bayberry fruits. Therefore, not only the detection efficiency of Dingao bayberry can be improved, but also the detection accuracy can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, without paying creative labor, other drawings obtained based on these drawings still belong to the scope of the present invention.

[0017] Figure 1 A flowchart of a method for detecting the maturity of Dingao bayberry based on improved Faster RCNN provided in an embodiment of the present invention; Figure 2Schematic diagram of Dingao bayberry photographed in different natural environments in a method for detecting the maturity of Dingao bayberry based on improved Faster RCNN provided by an embodiment of the present invention; wherein (a) is an unobstructed environment; (b) is an environment with multiple fruits overlapping and branches and leaves obstructing; (c) is an environment with weak light and branches and leaves obstructing; (d) is an environment with strong light and multiple fruits overlapping; Figure 3 A comparison diagram of the maturity of Dingao bayberry fruit in a Dingao bayberry maturity detection method based on improved Faster RCNN provided in an embodiment of the present invention; wherein (a) is immature; (b) is semi-mature; (c) is nearly mature; (d) is fully mature; Figure 4 A structural diagram of an improved Faster RCNN network model in a Dingao bayberry maturity detection method based on an improved Faster RCNN provided in an embodiment of the present invention; Figure 5 A structural diagram of the SE attention mechanism module in a method for detecting the maturity of Dingao bayberry based on an improved Faster RCNN provided in an embodiment of the present invention; Figure 6 A structural diagram of the FPN module in a method for detecting the maturity of Dingao bayberry based on an improved Faster RCNN provided in an embodiment of the present invention; Figure 7 A structural diagram of the deep fusion of the ConvNeXt-T module and the FPN module in a Dingao bayberry maturity detection method based on an improved Faster RCNN provided in an embodiment of the present invention; Figure 8 An original image of Dingao bayberry in a method for detecting the maturity of Dingao bayberry based on improved Faster RCNN provided in an embodiment of the present invention, and a detection result image using a traditional Faster RCNN model and the present invention; wherein (a) is the original image of Dingao bayberry; (b) is the detection result image of the traditional Faster RCNN model; (c) is the detection result image of the present invention; Fig. 9 A structural schematic diagram of a Dingao bayberry maturity detection system based on improved Faster RCNN provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings.

[0019] like Figure 1 As shown in the figure, a method for detecting the maturity of Dingao bayberry based on improved Faster RCNN is provided in an embodiment of the present invention, and the method comprises the following steps: Step S1, obtaining a Dingao bayberry image set, marking each image in the Dingao bayberry image set, and further dividing it into a training set, a validation set, and a test set; wherein the Dingao bayberry image set is composed of unripe images of Dingao bayberry, semi-ripe images of Dingao bayberry, nearly ripe images of Dingao bayberry, and fully ripe images of Dingao bayberry; The specific process is as follows: First, in order to ensure the diversity and comprehensiveness of the data, during the entire growth period of Dingao Bayberry, high-performance cameras were used to shoot multiple times at different distances, angles, lighting conditions, and densities (such as strong light, weak light, no obstruction, multiple fruit overlap, foliage obstruction, and other natural environments), and finally 757 sample images at different growth stages were obtained with a resolution of 4032×3024. Figure 2 Shown are images of Dingao Bayberry taken in different natural environments.

[0020] Secondly, during the growth process of Dingao Bayberry, the color and size of the fruit will change significantly over time. The fruit color will gradually change from turquoise to yellow-white, pink, red, and finally purple-red or even purple-black. At the same time, the fruit size will gradually increase and eventually reach a maximum value. Therefore, according to the actual growth and development stage and the changes in appearance color, the maturity of Dingao Bayberry can be divided into four categories: the fruit is small overall, the skin is all turquoise and there is no obvious red, which is immature; the fruit is of moderate size, and the skin turns yellow and red but is not completely red, which is semi-mature; the fruit is of moderate size, and the skin is all red but has not completely turned purple-black, which is nearly mature; the fruit is the largest and the skin is all purple-black, which is fully mature. Figure 3 Shown is a comparison chart of the maturity of Dingao bayberry fruit.

[0021] Next, in order to effectively filter out interference information and improve the quality of the collected images, the original images were preprocessed using mapping tools, and the above interference information was uniformly mosaicked. At the same time, in order to further improve the generalization ability of the model and effectively prevent overfitting due to insufficient data, the image was enhanced and amplified using an offline enhancement algorithm. Specifically, each image was subjected to a variety of image enhancement processes including random rotation, translation, left-right flipping, cropping, and scaling. In addition, the Mosaic data enhancement algorithm was applied to generate more diverse training samples. After these data enhancement operations, the number of original sample images was expanded from 779 to 4608, and the amplified data was divided into training set, validation set, and test set in a ratio of 7:2:1, which were used for model training, validation, and testing, respectively.

[0022] At the same time, the PASCAL VOC data format was used to annotate the Dingao bayberry fruits of different maturity in the image using LabelImg software to generate the corresponding XML file. Among them, unripe is marked as Unripe, half-ripe is marked as Half-ripe, near-ripe is marked as Near-ripe, and fully-ripe is marked as Ripe. A total of 49,078 Dingao bayberry fruit maturity labels were annotated. The number of Dingao bayberry annotations of different maturity in each data set is shown in Table 1.

[0023] Table 1 Step S2, based on the improved Faster RCNN network model, a maturity detection model of Dingao bayberry is constructed; wherein the improved Faster RCNN network model includes a feature extraction backbone network, an RPN network, a RoI pooling layer and a fully connected layer composed of a ConvNeXt-T module and an FPN module that introduces an SE attention mechanism; the ConvNeXt-T module is used to extract key features of the input image to form a four-layer feature map; the FPN module is used to perform multi-scale feature fusion on the four-layer feature maps output by the ConvNeXt-T module to form a multi-scale feature atlas; The specific process is as follows: first, the inventors found that Faster RCNN is an algorithm based on two-stage target detection, and its design significantly improves the detection accuracy and speed. Its network architecture mainly includes feature extraction backbone (Backbone), region proposal network (RPN network), region of interest pooling (RoI pooling layer) and fully connected layer. Among them, the feature extraction network uses a deep convolutional neural network (CNN) to extract rich feature representations from the input image, and the extracted features are used as inputs for the subsequent RPN network and RoI pooling layer; the RPN network uses anchor boxes to generate recommended regions on the feature map, and screens out high-confidence recommended regions and inputs them into the RoI pooling layer; the RoI pooling layer uniformly maps the candidate regions of different sizes generated by the RPN network to a fixed-size feature map, retaining important spatial and contextual information, and providing consistent input for classification and bounding box regression; the fully connected layer is processed by two parallel branches of classification and regression. The classification network predicts the category of each candidate region, and the regression network fine-tunes the bounding box of the candidate region to finally determine the location of the target. The Faster RCNN model combines the feature extraction capabilities of deep learning with a refined detection process, and can efficiently detect multiple target types in complex scenes.

[0024] In order to further improve the accuracy of target detection in complex environments, the Faster RCNN model is improved by introducing the ConvNeXt-T module of the SE attention mechanism and the FPN module to form a feature extraction backbone network. The specific structure is as follows: Figure 4 shown.

[0025] (1) ConvNeXt-T module. The traditional Faster RCNN uses VGG16 as the feature extraction backbone network. After multiple convolution and pooling operations, it is easy to cause the feature information of small targets to be lost, resulting in missed detection or false detection. Therefore, the original feature extraction backbone network is replaced with the ConvNeXt-T module, which is more suitable for complex scenes and small target detection, to better capture the detailed information of small targets, thereby improving the overall recognition performance of the model.

[0026] ConvNeXt-T is a network architecture in the ConvNeXt series, which is designed to balance computational efficiency and performance. The specific structure is shown in Table 2. Its overall network structure is similar to ResNet, including a downsampling module and a four-stage backbone structure consisting of ConvNeXt Block modules.

[0027] Among them, the downsampling module is located before the backbone structure (i.e., the starting stage of the network), which is used to receive the input image and downsample the features of the input image, and further reduce the spatial size of the downsampled feature map through pooling or stride convolution to increase the receptive field and extraction capability of the feature.

[0028] Among them, the backbone structure is divided into four stages; each stage includes a ConvNeXt Block module, and each ConvNeXt Block module contains a 1x1 convolution layer and a 3x3 grouped convolution layer; the Block ratio of each stage is 1:1:3:1, and the corresponding number of Blocks is 3, 3, 9, and 3, and the number of channels is 96, 192, 384, and 768 respectively; each stage performs feature extraction through a 4x or 2x downsampling rate, and integrates the SE attention mechanism to form the corresponding feature map.

[0029] It should be noted that the ConvNeXt Block module, as the core part of ConvNeXt-T, contains multiple convolution blocks and adopts an inverse bottleneck structure design (large in the middle and small at both ends). These convolution blocks can effectively avoid the loss of information features through appropriate channel organization and feature interaction, and further improve the expression ability and stability of the network.

[0030] Table 2 (2) SE attention mechanism. Dingao bayberry grows in a natural environment. Its fruits are often blocked by branches and leaves and are generally small in size, which results in insufficient detection accuracy of traditional detection models such as Faster RCNN. In order to improve the detection effect of the model in complex environments, the SE attention mechanism is integrated into the S1, S2, S3, and S4 modules in the ConvNeXt-T module. The SE attention mechanism recalibrates the channel weights of the feature map through compression and excitation operations, thereby enhancing the ability to extract key features and effectively alleviating the problem of missed detection of small targets. Figure 5 This is the network structure of the SE attention mechanism.

[0031] The SE module first performs global average pooling on the feature map of size HxWxC through a squeeze operation, compressing the spatial information of each channel into a 1x1xC feature vector. Then, through two fully connected layers and a Sigmoid activation function (Excitation operation), the weight of each channel is automatically learned and generated. These weights recalibrate the feature importance of each channel, allowing the model to focus more on the feature channels that are critical to target detection. Finally, each channel of the original feature map is multiplied by the generated weight to complete the channel recalibration. The introduction of the SE module in the feature fusion module enables the model to focus more on the features that are critical to fruit maturity detection, thereby improving the accuracy and robustness of detection in complex environments.

[0032] (3) FPN module. In traditional deep learning, after the image is processed by multiple layers of convolution and downsampling, the semantic information in the feature map becomes richer and richer, but its position and resolution are low, which makes the information of small-scale objects easily submerged or ignored. To address this problem, the feature pyramid network (FPN module) is introduced in the ConvNeXt-T module. Figure 6 As shown in the figure, the FPN module combines high-level semantic information with low-level detail information through multi-scale feature fusion, improving the ability to capture small-scale targets without significantly increasing the amount of calculation. The FPN module transmits high-level features through a top-down path and fuses them with low-level features through lateral connections, thereby generating a multi-scale feature map containing rich semantic and detail information. like Figure 7As shown in the figure, it is a structural diagram after deep fusion of ConvNeXt-T module and feature pyramid network (FPN module). Based on the four-layer feature map {C1, C2, C3, C4} output by ConvNeXt-T module as the backbone network, through top-down path design, the high-level feature map is upsampled and added element by element with the low-level feature map to achieve feature fusion. This process is carried out layer by layer, and finally a multi-scale feature map set {P1, P2, P3, P4, P5} containing rich semantic information and fine detail information is generated. This structure effectively solves the problem of feature transmission and fusion between different scales and levels, which helps the model capture more target detail information.

[0033] At this time, the evaluation indicators of the model include precision (P), recall (R), mean average precision (mAP), model size, and inference speed to evaluate the model performance. The specific calculation formula is as follows: (1); (2); (3); (4); Among them, in formulas (1) to (4), T P Indicates the number of samples correctly predicted as positive by the model; F N Indicates the number of positive samples that are mistakenly predicted as negative by the model; F P Indicates the number of negative samples that are incorrectly predicted as positive by the model; N is the total number of detection categories, AP i is the average precision of the i-th category.

[0034] Step S3: training the maturity detection model based on the training set, and further verifying and testing the maturity detection model based on the verification set and the test set to obtain a trained maturity detection model; The specific process is as follows: the training environment is Windows 10 (64-bit) operating system, the processor is Intel(R) Corei7-10700F, and the memory is 32GB. To meet the needs of high-performance computing, the graphics card is NVIDIA GeForce RTX 4090, equipped with 24GB video memory. The Python version is 3.8.10, and it is developed and compiled on PyCharm IDE. The deep learning framework uses PyTorch 2.1.1, and the GPU acceleration library is CUDA 12.1 and cuDNN 8.9.0.

[0035] The constructed training set and validation set are randomly shuffled and input into the maturity detection model for training. The input image resolution of the model is set to 600×600, the initial learning rate is 0.01, the iteration batch size is 16, and the total number of iterations (epochs) is 150. During the training process, the model's precision (P), recall rate (R), and loss value (loss) continue to change with the increase in the number of training times. The changes in these indicators directly reflect the training status of the entire model. The change curve of the model's loss value (loss) during training is shown in the figure below. Figure 7 As the number of training rounds (epochs) increases, the loss value of the model shows a continuous downward trend; when the number of training rounds reaches 120, the loss value begins to flatten out, and finally after 150 epochs of training, the model reaches a convergence state, thus obtaining a trained maturity detection model.

[0036] Step S4, obtaining the image to be tested of Dingao bayberry, and importing the image to be tested into the trained maturity detection model to obtain the maturity of the Dingao bayberry corresponding to the image to be tested as one of immature, semi-mature, nearly mature and fully mature.

[0037] The specific process is to obtain the image of Dingao Bayberry to be tested, and import it into the maturity detection model trained in step S3 to obtain the maturity of Dingao Bayberry corresponding to the image to be tested, and the result is one of immature, semi-mature, nearly mature and fully mature.

[0038] In this embodiment of the present invention, in order to verify the effectiveness of the maturity detection model in the target detection task, five groups of ablation experiments are designed, namely, the feature extraction network is ResNet50, the feature extraction network is replaced by the ConvNeXt-T module, the SE attention mechanism is introduced into the feature extraction network, the FPN module is integrated into the feature extraction network, and the improved model proposed by the present invention is shown in Table 3.

[0039] Table 3 1) The results of the five ablation experiments show that the proposed method can effectively improve the model's accuracy in detecting the maturity of Dingao Bayberry. Compared with the traditional Faster RCNN model, the three improvements increased the mean average precision (mAP) by 8.83%, 2.83%, and 4.03%, respectively. Overall, compared with the Faster RCNN model before the improvement, the improved algorithm proposed in this paper increased the mean average precision (mAP) by 17.27%, which can effectively improve the detection performance of Dingao Bayberry at different maturity levels.

[0040] 2) Compared with the traditional Faster RCNN model on the Dingao bayberry images in the test set, the detection effect of the present invention on four typical scenes, namely small targets, multiple targets overlapping occlusion, multiple targets overlapping occlusion in weak light, and multiple targets overlapping occlusion in strong light, is good, without obvious missed detection, false detection, inaccurate positioning, repeated detection, etc., and can realize the detection of different maturity of Dingao bayberry fruits with high precision and stability.

[0041] In order to more intuitively compare the detection effect of the present invention, four typical scenes including small targets, multiple targets overlapping and occluding, multiple targets overlapping and occluding in weak light, and multiple targets overlapping and occluding in strong light were selected from the Dingao Bayberry test set constructed above, and the traditional Faster RCNN model was compared with the present invention on the samples. The detection results are shown in the figure. Figure 8 shown.

[0042] from Figure 8 In (b), analysis shows that the traditional Faster RCNN model has average detection effects on small targets, multiple targets and different light intensities, and there are cases of missed detection. In small target detection, two small target fruits that are close to each other are easily identified as one target fruit, resulting in false detection; in multi-target detection, the detection effect on severely occluded targets and immature fruits blocked by leaves is poor, and the model fails to effectively extract and utilize the key feature information of Dingao Bayberry fruit, resulting in obvious missed detection problems; in multi-target detection under different light intensities, the model's detection accuracy is low, there are repeated detections, and there are also multiple missed detections and false detections. From Figure 8 In (c), the analysis shows that the detection effect of the proposed model is good under small targets, multiple targets and different light intensities, and there is no obvious missed detection, false detection, inaccurate positioning, repeated detection, etc. In summary, the improved Faster RCNN model can accurately and stably realize the maturity detection of Dingao bayberry fruit, and also shows good detection accuracy and robustness in complex environments.

[0043] like Fig. 9 As shown in the embodiment of the present invention, a Dingao waxberry maturity detection system based on improved Faster RCNN is provided, comprising: The image set construction unit 110 is used to obtain a Dingao bayberry image set, mark each image in the Dingao bayberry image set, and further divide it into a training set, a validation set, and a test set; wherein the Dingao bayberry image set is composed of unripe images of Dingao bayberry, semi-ripe images of Dingao bayberry, nearly ripe images of Dingao bayberry, and fully ripe images of Dingao bayberry; The maturity detection model construction unit 120 is used to construct a maturity detection model for Dingao bayberry based on an improved Faster RCNN network model; wherein the improved Faster RCNN network model includes a feature extraction backbone network, an RPN network, a RoI pooling layer, and a fully connected layer composed of a ConvNeXt-T module and an FPN module that introduce an SE attention mechanism; the ConvNeXt-T module is used to extract key features of an input image to form a four-layer feature map; the FPN module is used to perform multi-scale feature fusion on the four-layer feature maps output by the ConvNeXt-T module to form a multi-scale feature atlas; A maturity detection model training unit 130 is used to train the maturity detection model based on the training set, and further verify and test the maturity detection model based on the verification set and the test set to obtain a trained maturity detection model; The maturity detection unit 140 is used to obtain the image to be tested of Dingao Bayberry, and import the image to be tested into the trained maturity detection model to obtain the maturity of the Dingao Bayberry corresponding to the image to be tested as one of immature, semi-mature, nearly mature and fully mature.

[0044] The ConvNeXt-T module includes a downsampling module and a four-stage backbone structure consisting of ConvNeXt Block modules; The downsampling module is located before the backbone structure, and is used to receive an input image, downsample the features of the input image, and further reduce the spatial size of the downsampled feature map by pooling or stride convolution, thereby increasing the receptive field and extraction capability of the features; The backbone structure is divided into four stages; each stage includes a ConvNeXt Block module, and each ConvNeXt Block module contains a 1x1 convolution layer and a 3x3 grouped convolution layer; the Block ratio of each stage is 1:1:3:1, and the corresponding number of Blocks is 3, 3, 9, and 3, and the number of channels is 96, 192, 384, and 768 respectively; each stage performs feature extraction through a 4x or 2x downsampling rate, and integrates the SE attention mechanism to form a corresponding feature map.

[0045] Wherein, each image in the Dingao waxberry image set and the image to be tested are subjected to image mosaic processing and image enhancement processing.

[0046] The image enhancement processing includes random image rotation, image translation, left-right flipping, image cropping and image scaling.

[0047] Implementing the embodiments of the present invention has the following beneficial effects: The maturity detection model of the present invention is constructed based on an improved Faster RCNN network model. Since a ConvNeXt-T module and an FPN module of an SE attention mechanism are introduced into the improved Faster RCNN network model to form a feature extraction backbone network, the ConvNeXt-T module can be used to improve recognition performance in a complex environment, the SE attention mechanism is introduced to enhance the network's learning ability for different maturity characteristics of Dingao bayberry fruits, and the FPN module (feature pyramid network) is combined to improve the detection ability of small-sized Dingao bayberry fruits. Therefore, not only the detection efficiency of Dingao bayberry can be improved, but also the detection accuracy can be improved.

[0048] It is worth noting that in the above system embodiment, the various system modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the various functional modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0049] A person skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, such as ROM / RAM, disk, CD-ROM, etc.

[0050] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A method for detecting the maturity of Dingao bayberry based on improved Faster RCNN, characterized in that: The method comprises the following steps: Obtain a Dingao bayberry image set, and mark each image in the Dingao bayberry image set, and further divide it into a training set, a validation set, and a test set; wherein the Dingao bayberry image set is composed of unripe images of Dingao bayberry, semi-ripe images of Dingao bayberry, nearly ripe images of Dingao bayberry, and fully ripe images of Dingao bayberry; Based on the improved Faster RCNN network model, a maturity detection model for Dingao bayberry is constructed; wherein the improved Faster RCNN network model includes a feature extraction backbone network composed of a ConvNeXt-T module and an FPN module that introduces the SE attention mechanism, an RPN network, a RoI pooling layer, and a fully connected layer; the ConvNeXt-T module is used to extract key features of the input image to form a four-layer feature map; the FPN module is used to perform multi-scale feature fusion on the four-layer feature maps output by the ConvNeXt-T module to form a multi-scale feature atlas; Based on the training set, the maturity detection model is trained, and further based on the validation set and the test set, the maturity detection model is validated and tested to obtain a trained maturity detection model; The image to be tested of Dingao bayberry is obtained, and the image to be tested is imported into the trained maturity detection model to obtain the maturity of the Dingao bayberry corresponding to the image to be tested, which is one of immature, semi-mature, nearly mature and fully mature.

2. The Dingao waxberry maturity detection method based on improved Faster RCNN as claimed in claim 1, characterized in that: The ConvNeXt-T module includes a downsampling module and a four-stage backbone structure consisting of ConvNeXt Block modules; The downsampling module is located before the backbone structure, and is used to receive an input image, downsample the features of the input image, and further reduce the spatial size of the downsampled feature map by pooling or stride convolution, thereby increasing the receptive field and extraction capability of the features; The backbone structure is divided into four stages; each stage includes a ConvNeXt Block module, and each ConvNeXt Block module contains a 1x1 convolution layer and a 3x3 grouped convolution layer; the Block ratio of each stage is 1:1:3:1, and the corresponding number of Blocks is 3, 3, 9, and 3, and the number of channels is 96, 192, 384, and 768 respectively; each stage performs feature extraction through a 4x or 2x downsampling rate, and integrates the SE attention mechanism to form a corresponding feature map.

3. The Dingao waxberry maturity detection method based on improved Faster RCNN as claimed in claim 2, characterized in that: Each image in the Dingao waxberry image set and the image to be tested are subjected to image mosaic processing and image enhancement processing.

4. The Dingao waxberry maturity detection method based on improved Faster RCNN as claimed in claim 3, characterized in that: The image enhancement processing includes random image rotation, image translation, left-right image flipping, image cropping and image scaling.

5. A Dingao waxberry maturity detection system based on improved Faster RCNN, characterized in that: include: An image set construction unit is used to obtain a Dingao bayberry image set, mark each image in the Dingao bayberry image set, and further divide it into a training set, a validation set, and a test set; wherein the Dingao bayberry image set is composed of unripe images of Dingao bayberry, semi-ripe images of Dingao bayberry, nearly ripe images of Dingao bayberry, and fully ripe images of Dingao bayberry; A maturity detection model construction unit is used to construct a maturity detection model for Dingao bayberry based on an improved Faster RCNN network model; wherein the improved Faster RCNN network model includes a feature extraction backbone network, an RPN network, a RoI pooling layer, and a fully connected layer composed of a ConvNeXt-T module and an FPN module that introduces an SE attention mechanism; the ConvNeXt-T module is used to extract key features of an input image to form a four-layer feature map; the FPN module is used to perform multi-scale feature fusion on the four-layer feature maps output by the ConvNeXt-T module to form a multi-scale feature atlas; A maturity detection model training unit, used to train the maturity detection model based on the training set, and further verify and test the maturity detection model based on the verification set and the test set to obtain a trained maturity detection model; The maturity detection unit is used to obtain the image to be tested of Dingao bayberry, and import the image to be tested into the trained maturity detection model to obtain the maturity of the Dingao bayberry corresponding to the image to be tested as one of immature, semi-mature, nearly mature and fully mature.

6. The Dingao waxberry maturity detection system based on improved Faster RCNN as claimed in claim 5, characterized in that: The ConvNeXt-T module includes a downsampling module and a four-stage backbone structure consisting of ConvNeXt Block modules; The downsampling module is located before the backbone structure, and is used to receive an input image, downsample the features of the input image, and further reduce the spatial size of the downsampled feature map by pooling or stride convolution, thereby increasing the receptive field and extraction capability of the features; The backbone structure is divided into four stages; each stage includes a ConvNeXt Block module, and each ConvNeXt Block module contains a 1x1 convolution layer and a 3x3 grouped convolution layer; the Block ratio of each stage is 1:1:3:1, and the corresponding number of Blocks is 3, 3, 9, and 3, and the number of channels is 96, 192, 384, and 768 respectively; each stage performs feature extraction through a 4x or 2x downsampling rate, and integrates the SE attention mechanism to form a corresponding feature map.

7. The Dingao waxberry maturity detection system based on improved Faster RCNN as claimed in claim 6, characterized in that: Each image in the Dingao waxberry image set and the image to be tested are subjected to image mosaic processing and image enhancement processing.

8. The Dingao waxberry maturity detection system based on improved Faster RCNN as claimed in claim 7, characterized in that: The image enhancement processing includes random image rotation, image translation, left-right image flipping, image cropping and image scaling.

Images