Method and system for citrus detection under complex background based on improved deep learning

Abstract

The present application relates to a citrus detection method and system based on improved deep learning in a complex background, the method comprising, collecting citrus images in an orchard in a complex background to obtain an initial data set; preprocessing the initial data set to obtain a first data set; labeling and format converting the first data set to obtain a target data set; constructing an improved YOLOv7-tiny network model as a citrus detection network model with the YOLOv7-tiny network model as a deep learning benchmark model; training and evaluating the citrus detection network model; obtaining the optimal weight of the citrus detection network model and testing the citrus detection network model; obtaining the test result through the citrus detection model and calculating the precision and recall rate based on the intersection over union (IOU) to further judge the accuracy of the detection. The improved YOLOv7-tiny network model of the present application can effectively detect small and medium-sized citrus targets, improve the detection accuracy and speed of the model, and at the same time, take into account the lightweight of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a citrus detection method and system based on improved deep learning in a complex background. BACKGROUND

[0002] For the field of fruit recognition, past citrus detection has relied heavily on traditional target detection methods. Traditional methods mainly rely on manual features for target detection, which is time-consuming and labor-intensive, and in complex environments, it is difficult to accurately identify citrus when facing problems such as branch and leaf shading, light changes, etc., greatly hindering the intelligent development of the citrus industry.

[0003] In the field of deep learning, the YOLOv7-tiny algorithm provides an innovative solution for citrus identification, especially in complex backgrounds such as different sparsity of orchard environments, changes in lighting conditions, and different maturity stages of fruits. With its powerful feature extraction capability and efficient detection speed, YOLOv7-tiny can flexibly adapt to changes in various environments and adapt to the detection needs of small and medium-sized targets while ensuring high detection accuracy. However, the current YOLOv7-tiny algorithm still has problems of redundant calculation and parameter quantity, and the inference speed is slow, which makes it difficult to meet the needs of real-time decision-making in orchards.

[0004] Therefore, providing a new citrus detection method and system based on improved deep learning in a complex background can improve the application efficiency of deep learning in the field of agricultural intelligence, promote the realization of real-time and accurate orchard management and decision-making systems, and thus bring higher automation level and more accurate crop management to agricultural production. SUMMARY

[0005] Based on the above problems existing in the prior art, the purpose of the embodiments of the present application is to provide a citrus detection method and system based on improved deep learning in a complex background, which can effectively detect small and medium-sized citrus targets, improve the detection accuracy and speed of the model while taking into account the lightweight of the model.

[0006] To achieve the above purpose, the technical solution adopted by the present application is: a citrus detection method based on improved deep learning in a complex background, comprising:

[0007] S1, collecting complex background citrus images in an orchard to obtain an initial data set;

[0008] S2, preprocessing the initial data set to obtain a first data set;

[0009] S3, labeling and format converting the first data set to obtain a target data set;

[0010] S4, taking the YOLOv7-tiny network model as a deep learning benchmark model, an improved YOLOv7-tiny network model is constructed as a citrus detection network model; wherein the improved YOLOv7-tiny network model includes replacing the Conv module in all CBS modules in the backbone network with a PConvBS module; the ELAN module in the backbone network is replaced with a Faster-PELAN module in a feature fusion manner; the ELAN-H module is used in the neck network for multi-scale feature fusion; the backbone network and the detection head are optimized, including reducing the number of down-sampling times of the backbone network and removing the corresponding large target detection head; the detection head uses alphaShape-IoU Loss as the loss function;

[0011] S5, training and evaluating the citrus detection network model;

[0012] S6, obtaining the optimal weight of the citrus detection network model and testing the citrus detection network model;

[0013] S7, obtaining the test result through the citrus detection model and calculating the precision and recall based on the intersection over union (IOU) to judge the accuracy of the detection.

[0014] Further, in S2, the pre-processing of the initial data set includes filtering, data enhancement, and resolution formatting of the initial data set.

[0015] Further, in S2, the pre-processing of the initial data set to obtain the first data set includes the following steps:

[0016] Step S21, filtering the initial data set, including deleting images that are unclear due to factors such as shooting, light, and device stability, and deleting citrus image data that lack key features;

[0017] Step S22, data enhancement includes color space transformation and direction reversal of images;

[0018] Step S23, resolution formatting includes resolution formatting processing of the initial data set, and selecting a suitable standard resolution according to the requirements of model training and the actual situation of hardware resources.

[0019] Further, in S3, the specific content of labeling the first data set includes labeling categories, labeling center points, and labeling width and height information, and the format conversion of the first data set includes reorganizing and extracting the labeling information in the xml file to obtain a txt format label file as the target data set.

[0020] Further, the floating point operation number and memory access amount calculation formula of the PConvBS module is as follows:

[0021]

[0022] Wherein, h is the height of the feature map, w is the width of the feature map, k is the size of the convolution kernel, c p is the output channel number, in the formula, c p / c=1 / 4.

[0023] Further, the Faster-PELAN module is composed of a PConvBS module, a DWConvBS module, a Concat module and a CBS module; the Faster-PELAN module has two paths, the first path will pass through the PConvBS module and the DWConvBS module for fast sampling, and the second path will pass through the

[0024] PConvBS module for sampling; the feature map after sampling of a specific layer on the first path will be spliced with the feature map after sampling of the second path, and finally a convolution layer is used to complete the feature aggregation process; the

[0025] DWConvBS module is sequentially composed of a DWConvBS module, a BatchNormalization layer and a SiLU activation function.

[0026] Further, the calculation formula of the loss function αShape-IoU Loss is as follows:

[0027] L αShape-IoU =α(1-IoU)+(1-α)distance shape +0.5×Ω shape

[0028] Wherein, L αShape-IoU is the loss function, α is the weight parameter, IoU is the intersection over union, distance shape is the shape distance, Ω shape is the shape complexity related term;

[0029] The calculation formula of the intersection over union IoU is as follows:

[0030]

[0031] Wherein, IoU is the intersection over union, B is the predicted bounding box, B gt is the real bounding box;

[0032] The intersection over union IoU is used to measure the overlapping degree of the predicted box and the real box, the value of IoU is between 0 and 1, the larger the value is, the higher the coincidence degree of the predicted box and the real box is.

[0033] shape distance distance shape The calculation formula is:

[0034]

[0035] Wherein, distance shape is the shape distance, x c and y c are the coordinates of the center of the prediction box, and are the coordinates of the center of the real box, c is a constant, HH and WW are coefficients related to the width and height of the box, w gt is the width of the real box, h gt is the height of the real box, (w gt ) scale is the new width value obtained by scaling the width of the real box, (h gt ) scale is the new height value obtained by scaling the height of the real box.

[0036] Shape complexity related term Ω shape The calculation formula is:

[0037]

[0038] Wherein, Ω shape is the shape complexity related term, w t is the shape of the box, and θ is a fixed parameter.

[0039] The citrus detection system based on improved deep learning in complex background is applied to the above-mentioned citrus detection method based on improved deep learning in complex background, and the system comprises:

[0040] A data set acquisition module is configured to acquire complex background citrus images in an orchard to obtain an initial data set.

[0041] A data set preprocessing module is configured to preprocess the initial data set to obtain a first data set.

[0042] A data set format conversion module is configured to label and convert the format of the first data set to obtain a target data set.

[0043] An improved modeling module is used to construct an improved YOLOv7-tiny network model as a citrus detection network model based on a YOLOv7-tiny network model as a deep learning benchmark model; wherein the improved YOLOv7-tiny network model comprises replacing the Conv module in all CBS modules in the backbone network with a PConvBS module; the ELAN module in the backbone network is replaced with a Faster-PELAN module in a feature fusion manner; the ELAN-H module is used in the neck network for multi-scale feature fusion; the backbone network and the detection head are optimized, including reducing the number of down-sampling times of the backbone network and removing the corresponding large target detection head; the detection head uses alphaShape-IoU Loss as the loss function;

[0044] A model training and evaluation module is used to train and evaluate the citrus detection network model.

[0045] A model testing module is used to obtain the optimal weight of the citrus detection network model and test the citrus detection network model.

[0046] A detection result generation module is used to obtain the test result through the citrus detection model and calculate the precision and recall based on the intersection over union (IOU) to further judge the accuracy of the detection.

[0047] The embodiment of the present application also provides a network side server, comprising:

[0048] At least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above-mentioned citrus detection method based on improved deep learning in a complex background.

[0049] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the above-mentioned citrus detection method based on improved deep learning in a complex background.

[0050] The beneficial effects of the present application are: the improved deep learning-based citrus detection method in complex background of the present application, including collecting complex background citrus images in the orchard to obtain an initial data set; preprocessing the initial data set to obtain a first data set; labeling and format converting the first data set to obtain a target data set; taking the YOLOv7-tiny network model as the deep learning benchmark model to construct an improved YOLOv7-tiny network model as the citrus detection network model; wherein the improved YOLOv7-tiny network model includes replacing the Conv module in all CBS modules in the backbone network with a PConvBS module; replacing the ELAN module in the backbone network with a Faster-PELAN module in a feature fusion manner; using the ELAN-H module in the neck network for multi-scale feature fusion; optimizing the backbone network and the detection head, including reducing the downsampling times of the backbone network and removing the corresponding large target detection head; using alphaShape-IoU Loss as the loss function of the detection head; training and evaluating the citrus detection network model; obtaining the optimal weight of the citrus detection network model and testing the citrus detection network model; obtaining the test result through the citrus detection model and calculating the precision and recall based on the intersection over union IOU to further judge the accuracy of the detection. The improved deep learning-based citrus detection method in complex background of the present application replaces the Conv module in the backbone network with the PConvBS module, reduces redundant feature extraction and model parameter quantity, optimizes memory usage and calculation amount, improves feature extraction efficiency, replaces the ELAN module with the Faster-PELAN module, enhances multi-scale feature aggregation capability, realizes efficient feature sampling and aggregation; the neck network uses the ELAN-H module for multi-scale feature fusion, reduces the calculation amount and parameter quantity, improves the inference speed, and is suitable for real-time application; the backbone network and the detection head are optimized, the downsampling times are reduced, and the large target detection head is removed, which reduces the parameter quantity, improves the inference speed, and can effectively detect small and medium-sized citrus targets; the detection head uses alphaShape-IoU Loss as the loss function, which solves the problem of inaccurate positioning of small fruit targets; the format of the model weight file is converted, which reduces the memory occupation and calculation amount; the memory access is optimized, and technologies such as cache and data prefetching are used to improve the data processing speed. BRIEF DESCRIPTION OF DRAWINGS

[0051] The present application will be further described below in conjunction with the drawings and examples.

[0052] In the drawings:

[0053] Figure 1 The flowchart of the improved deep learning-based citrus detection method in complex background provided for the first embodiment of the present application;

[0054] Figure 2A structural schematic diagram of the improved YOLOv7-tiny provided for the first embodiment of the present application;

[0055] Figure 3 A structural schematic diagram of the PConvBS module provided for the first embodiment of the present application;

[0056] Figure 4 A structural schematic diagram of the DWConvBS module provided for the first embodiment of the present application;

[0057] Figure 5 A structural schematic diagram of the Faster-PELAN module provided for the first embodiment of the present application;

[0058] Figure 6 A structural schematic diagram of the ELAN-H module provided for the first embodiment of the present application;

[0059] Figure 7 A comparison chart of algorithm detection effects under different light intensities provided for the first embodiment of the present application;

[0060] Figure 8 A comparison chart of algorithm detection effects under different sparseness provided for the first embodiment of the present application;

[0061] Figure 9 A comparison chart of algorithm detection effects in an immature state provided for the first embodiment of the present application;

[0062] Figure 10 A module schematic diagram of the citrus detection system under a complex background based on the improved deep learning provided for the second embodiment of the present application;

[0063] Figure 11 A structural schematic diagram of a network side server provided according to the third embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the present application will be described clearly and completely below with reference to the drawings. Obviously, the described embodiments are some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0065] First embodiment:

[0066] The first embodiment of the present application provides a citrus detection method based on improved deep learning in a complex background, comprising: collecting citrus images in an orchard to obtain an initial data set; preprocessing the initial data set to obtain a first data set; labeling and format converting the first data set to obtain a target data set; taking a YOLOv7-tiny network model as a deep learning benchmark model to construct an improved YOLOv7-tiny network model as a citrus detection network model; wherein the improved YOLOv7-tiny network model comprises: replacing the Conv module in all CBS modules in the backbone network with a PConvBS module; replacing the ELAN module in the backbone network with a Faster-PELAN module in a feature fusion manner; using an ELAN-H module in the neck network for multi-scale feature fusion; optimizing the backbone network and the detection head, including reducing the number of downsampling times of the backbone network and removing the corresponding large target detection head; using alphaShape-IoU Loss as the loss function of the detection head; training and evaluating the citrus detection network model; obtaining the optimal weight of the citrus detection network model and testing the citrus detection network model; obtaining the test result through the citrus detection model and calculating the precision and recall based on the intersection over union (IOU) to judge the accuracy of the detection. The citrus detection method based on improved deep learning in a complex background can effectively detect small and medium-sized citrus targets, improve the detection accuracy and speed of the model, and take into account the lightweight of the model.

[0067] The implementation details of the citrus detection method based on improved deep learning in a complex background of the present embodiment will be described in detail below. The following content is only provided for the implementation details for easy understanding, and is not necessary for implementing the present solution. The specific process of the present embodiment is as shown in Figure 1 The present embodiment is applied to a citrus detection system based on improved deep learning in a complex background.

[0068] S1, collecting citrus images in a complex background in an orchard to obtain an initial data set.

[0069] Specifically, professional shooting equipment is used to shoot the citrus in the orchard from different angles, distances, and different lighting conditions, etc., aiming to obtain as many images as possible covering various shapes and different growth stages of the citrus; and the shooting covers different areas of the orchard, including places with different elevations, positions close to the edges of the orchard, and positions in the center of the orchard, etc., so as to accumulate a large amount of rich and diverse image data, ensuring that these data can truly reflect the overall appearance of the citrus in the actual orchard environment.

[0070] Due to the existence of various complex backgrounds in the orchard, for example, some citrus is grown in a place with dense branches and leaves, and the background presents a complex situation of interlaced dense leaves; some citrus may have a lot of weeds around, forming a complex background containing weeds; some citrus may be near the passageway of the orchard, and its background contains elements such as irrigation facilities and tools for fruit farmers to work. Based on these different complex background situations, the collected citrus image data is classified and arranged, and images with similar complex background characteristics are classified into the same category to construct an initial data set under different complex backgrounds.

[0071] Step S2, pre-processing the initial data set to obtain a first data set.

[0072] Specifically, the pre-processing of the initial data set includes screening, data enhancement and resolution formatting of the initial data set.

[0073] Step S21, screening the initial data set, including deleting images that are blurred due to factors such as shooting, light, and device stability, and deleting citrus image data that lacks key features.

[0074] Step S22, data enhancement includes color space transformation and direction reversal of images. Through color space transformation of images, the color features of citrus can be highlighted, such as in the HSV space, it is easier to separate the orange region of mature citrus from the green region of background leaves by threshold segmentation, which provides convenience for fruit recognition and positioning; in addition, data in different color spaces can be used as supplementary information to enable the model to learn the feature performance of citrus under multiple color description systems, enhance its adaptability to color changes, and improve its robustness. Image direction reversal includes horizontal and vertical reversal. By reversing the direction of the image, the data diversity can be further enriched, and the model can learn more pose changes of citrus, improving its generalization ability.

[0075] Step S23, resolution formatting includes resolution formatting processing of the initial data set, and according to the requirements of model training and the actual situation of hardware resources, a suitable standard resolution is selected, such as common 224x224 pixels or 512x512 pixels, etc. Through image scaling algorithm, the original images with different resolutions are uniformly adjusted to the set standard resolution, ensuring the consistency of the images in the data set in size, which is convenient for storage and management, optimizes the model training process, improves the overall efficiency, and at the same time, within a reasonable range, guarantees the image details, meets the requirements of the model for citrus feature extraction.

[0076] Step S3, labeling and format conversion of the first data set to obtain a target data set.

[0077] Specifically, the specific content of labeling the first data set includes labeling categories, labeling center points, and labeling width and height information. Citrus has growth stages and states, etc. When labeling categories, the category to which each citrus target belongs needs to be determined according to pre-set classification standards. For example, citrus can be divided into categories such as mandarin, navel orange, ponkan, etc. according to common citrus varieties; or into categories such as unripe, semi-ripe, and ripe according to the maturity of the citrus.

[0078] In order to accurately determine the position of the citrus in the image, the center point coordinates of each citrus target need to be labeled. Usually, the upper left corner of the image is taken as the coordinate origin (0, 0), the horizontal right direction is taken as the positive direction of the x-axis, and the vertical downward direction is taken as the positive direction of the y-axis. Through a professional labeling tool, the labeling personnel will accurately find the coordinate value (x, y) corresponding to the approximate geometric center position of the citrus fruit. As an example, the center point coordinates of a citrus in the image can be (120, 80). This coordinate information is very important for subsequent calculation of the relative position of the citrus and other objects, judgment of the layout of the citrus in the image, etc., and can help the model accurately lock the region where the citrus is located.

[0079] The width and height of the labeled citrus target are also measured in pixels. The maximum length of the citrus fruit in the horizontal direction is measured as the width along the x-axis direction, and the maximum length of the citrus fruit in the vertical direction is measured as the height along the y-axis direction. As an example, the width of a certain citrus is labeled as 50 pixels, and the height is labeled as 40 pixels. These size information, combined with the center point coordinates, can completely outline the specific range of the citrus in the image, so that the model can clearly obtain which pixel regions in the image belong to the citrus target to be concerned, thereby performing accurate feature extraction and analysis.

[0080] The format conversion of the first data set includes reorganizing and extracting the labeling information in the xml file to obtain a txt format label file as the target data set. For each citrus target, its category information is converted into the corresponding digital number according to the pre-set category index table (as an example, citrus corresponds to number 0, navel orange corresponds to number 1, etc.), and then the center point coordinates, width and height information are numerically converted according to a certain proportional relationship. These converted information is written into a txt file in a specific order, with each row representing the labeling information of a citrus target, and different numerical values are separated by spaces or other symbols. In this way, a txt format label file is generated as the target data set.

[0081] Step S4, using the YOLOv7-tiny network model as a deep learning benchmark model to construct an improved YOLOv7-tiny network model as a citrus detection network model.

[0082] Specifically, the existing YOLOv7-tiny network model mainly consists of a backbone network, a neck network and a detection head. Among them, the backbone network acts as a feature extractor, which extracts representative and discriminative features of different levels from the input image through a series of convolution, pooling and other operations; the neck network acts as a bridge between the backbone network and the detection head, which receives the feature maps of different levels extracted by the backbone network, and then integrates these features through feature fusion operations such as splicing, addition, etc., while further exploiting and utilizing context information, so that the features passed to the detection head are more comprehensive and targeted, which helps to improve the accuracy and robustness of detection; the detection head as the last part of the network, uses the integrated features of the neck network to generate the output of the network, that is, to predict the category and position of the objects in the image, output information such as the probability of each object belonging to different categories, the bounding box coordinates of the object in the image, etc., so as to realize the detection of objects in the image.

[0083] The detailed structure of the network model improved based on YOLOv7-tiny is shown in Figure 2 The specific steps are as follows:

[0084] Step S41, replace the Conv module in all CBS modules in the backbone network with the PConvBS module.

[0085] Specifically, as shown in Figure 3 , replace the Conv module in all CBS modules in the backbone network with the PConvBS module that only convolves part of the channels, to reduce the extraction of redundant features, reduce the parameter amount of the model, and improve the feature extraction efficiency.

[0086] The PConvBS module splits the input feature map into multiple sub-feature maps and assigns independent convolution kernels to each sub-feature map for processing, and then only convolves part of the channels. Unlike traditional convolution methods, the PConvBS module only extracts spatial features from part of the channels of the input feature map while keeping the other channels unchanged, thereby effectively reducing the amount of calculation and the extraction of redundant features. For continuous or regular memory access, the PConvBS module calculates the first or last continuous channel as a representative of the entire feature map, thereby optimizing memory usage and reducing unnecessary computational complexity. Therefore, the PConvBS module can significantly reduce the computational cost while improving the feature extraction efficiency when extracting spatial features. The number of floating point operations and memory access of the PConvBS module are calculated as follows:

[0087]

[0088] where h is the height of the feature map, w is the width of the feature map, k is the size of the convolution kernel, and c is the number of channels.p is the number of output channels. In the formula, c p / c=1 / 4.

[0089] As an example, in order to ensure the continuity of memory, the number of continuous output channels in the front or rear section is selected to represent the entire feature map. Still assuming that the number of input and output channels is the same, the number of floating point operations of the PConvBS module is (i.e. the convolution kernel and the input mapping Figure 1 The number of window calculations is k 2 If c p / c=1 / 4, then the number of floating point operations of the PConvBS module is only 1 / 16 of the ordinary convolution, and the memory access amount of the PConvBS module is If c p / c=1 / 4, then the memory access amount will be 1 / 4 of the ordinary convolution.

[0090] Further, the PConvBS module is sequentially composed of a PConv module, a BatchNormalization layer and a SiLU activation function.

[0091] Step S42, the ELAN module in the backbone network is replaced by the Faster-PELAN module in a feature fusion manner.

[0092] Specifically, as shown in Figure 5 , the original ELAN module is replaced by the feature fusion module Faster-PELAN, and the Faster-PELAN module is embedded into the main feature extraction network of the model, so as to enhance the feature aggregation ability of the network at multiple scales and better focus on the fruit target. The Faster-PELAN module is composed of multiple functional modules, and more efficient feature sampling and aggregation are realized through feature fusion of two different paths.

[0093] Among them, the Faster-PELAN module is composed of a PConvBS module, a DWConvBS module, a Concat module and a CBS module. The Faster-PELAN module has two paths, the first path will pass through the PConvBS module and the DWConvBS module for fast sampling, and the second path will pass through the PConvBS module for sampling; the feature map after sampling of the specific layer on the first path will be spliced with the feature map after sampling of the second path, and finally a convolution layer is used to complete the feature aggregation process.

[0094] Further, as shown in Figure 4 ​As shown, the DWConvBS module is sequentially composed of a DWConv module, a BatchNormalization layer and a SiLU activation function.

[0095] Step S43, multi-scale feature fusion is performed in the neck network by using the ELAN-H module.

[0096] Specifically, as shown in Figure 6 As shown, the ELAN-H module in the neck network reduces the amount of calculation and the amount of parameters, reduces memory consumption, and improves inference speed by using a simplified feature aggregation method and a more efficient convolution strategy (such as depthwise separable convolution), so that the improved YOLOv7-tiny network model maintains high detection accuracy while being more suitable for real-time applications, especially in embedded devices and resource-limited environments; the ELAN-H module can achieve a better balance between efficiency and accuracy, improving the target detection performance.

[0097] Step S44, the backbone network and the detection head are optimized, including reducing the downsampling times of the backbone network and removing the corresponding large target detection head.

[0098] Specifically, the downsampling times of the backbone network are reduced from five to four, the last layer MP module and the ELAN module in the backbone network are pruned, and the corresponding large target detection head is removed, reducing the overall parameter amount of the network model, improving the model inference speed and being able to effectively detect citrus as a small target.

[0099] Step S45, the detection head uses αShape-IoU Loss as the loss function.

[0100] Specifically, in the detection and recognition of citrus fruits, due to factors such as target overlap, occlusion and variable object shape, the existing detection head has the problem of poor detection performance. The detection head uses αShape-IoU Loss as the loss function, which can solve the problem of inaccurate positioning of small target positions in fruit targets.

[0101] The calculation formula of the loss function αShape-IoU Loss is:

[0102] L αShape-IoU = α(1-IoU) + (1-α)distance shape + 0.5 × Ω shape

[0103] Wherein, L αShape-IoU is the loss function, α is the weight parameter, IoU is the intersection over union, distance shape is the shape distance, Ω shape is the shape complexity related term.

[0104] Further, the calculation formula of the intersection over union IoU is:

[0105]

[0106] wherein, IoU is the intersection over union, B is the predicted bounding box, B gt is the real bounding box.

[0107] The IoU is used to measure the overlapping degree of the predicted box and the real box, the value of the IoU is between 0 and 1, the greater the value, the higher the coincidence degree of the predicted box and the real box.

[0108] Further, the calculation formula of the shape distance distance shape is:

[0109]

[0110] wherein, distance shape is the shape distance, x c and y c are the coordinates of the center of the predicted box, and are the coordinates of the center of the real box, c is a constant, HH and WW are coefficients related to the width and height of the box, w gt is the width of the real box, h gt is the height of the real box, (w gt ) scale is the new width value obtained by scaling the width of the real box, (h gt ) scale is the new height value obtained by scaling the height of the real box.

[0111] Further, the calculation formula of the shape complexity related term Ω shape is:

[0112]

[0113] wherein, Ω shape is the shape complexity related term, w t is the shape of the box, and θ is a fixed parameter.

[0114] Step S5, training and evaluating the citrus detection network model.

[0115] Specifically, the target data set is divided into a training set, a validation set, and a test set. Training the citrus detection network model includes training the improved YOLOv7-tiny citrus detection network model using the training set. During the training process, appropriate training parameters are set, such as the learning rate, the number of iterations, the batch size, etc. The initial learning rate can be set to a small value, and a learning rate decay strategy is used to gradually reduce the learning rate as the training progresses, to prevent the model from missing the optimal solution due to a too large learning rate in the early stage of training, and to continuously optimize the model parameters in the later stage of training. The batch size can be set to 16 or 32, etc. according to the hardware resources and data characteristics, to ensure that the model can learn the features of different samples in one iteration, while not being affected by memory overflow and other problems that affect the training process. The number of iterations can be adjusted according to the performance of the model on the validation set, and the training is stopped when the loss function value on the validation set no longer decreases significantly or overfitting is observed.

[0116] Further, an optimizer such as stochastic gradient descent or adaptive moment estimation is used to update the model parameters. In each iteration, the training samples are input into the model, the model calculates the predicted results through forward propagation, and then the loss value is calculated through the loss function aShape-IoU Loss according to the difference between the predicted results and the true labels. Then, the optimizer calculates the gradient through back propagation according to the loss value, and updates the weight and bias parameters of the model, so that the model gradually reduces the loss value and improves the detection accuracy in continuous training.

[0117] Evaluating the citrus detection network model includes periodically evaluating the model during the training process using the validation set, and the main evaluation indicators include the average precision (mAP), the recall rate, the precision rate, etc. The average precision (mAP) is an important indicator to measure the performance of the target detection model, which considers the detection accuracy and recall rate of different categories of citrus, and can comprehensively reflect the detection performance of the model at each threshold. By calculating the intersection over union (IoU) of the predicted box and the true box, and according to the set IoU threshold (determining the correctness of the prediction, the area under the precision-recall curve of different categories of citrus is calculated to obtain the mAP value. The recall rate reflects the proportion of the correctly detected citrus targets to the actual citrus targets, and the precision rate represents the proportion of the citrus targets that are truly citrus among the predicted citrus targets. Through the analysis of these indicators, the performance of the model in different aspects can be understood, the problems of the model can be found in time, and the training process can be adjusted.

[0118] Step S6, obtaining the optimal weight of the citrus detection network model and testing the citrus detection network model.

[0119] Specifically, the best model weight is determined in the training process in step S5. Since the previous step involves evaluating and comparing the model weights of different training stages, the loss function value, precision, recall, etc. on the validation set are comprehensively judged. Once the optimal weight is determined, the trained model is comprehensively tested using the weight. The test set should cover various complex backgrounds, different light conditions, different citrus varieties and growth stage image samples to ensure that the generalization ability of the model in the actual application scene is fully tested. During the testing process, the prediction results of the model for each sample are recorded in detail, including the predicted citrus category, bounding box position and corresponding confidence score, providing sufficient data support for subsequent result analysis.

[0120] Step S7, get the test results through the citrus detection model and calculate the precision and recall based on the intersection over union (IOU) to judge the accuracy of the detection.

[0121] Specifically, for the test results, according to the standard evaluation process of target detection, the intersection over union (IOU) of the predicted box and the label bounding box is used to judge the accuracy of the detection. For each prediction result, the IOU value with the true label is calculated accurately. When the IOU is greater than the pre-set threshold and the category prediction is correct, it is marked as a true positive T p ; if the intersection over union (IOU) is greater than the threshold but the category is incorrect, it is judged as a false positive T p ; and when there is no detection box on the target, it is defined as a missed detection F n . The precision and recall calculation formula is:

[0122]

[0123] Where P is the precision, T p is the true positive, F p is the false positive, R is the recall, and F n is the missed detection.

[0124] During the calculation process, the classification of each sample is carefully checked to ensure that the precision and recall are calculated accurately and correctly, so that the performance of the model in detecting citrus in complex backgrounds can be objectively evaluated.

[0125] As an example, for the trained citrus detection network model, its weight file is format-converted to adapt to the running environment of the embedded development board. During the conversion process, emphasis is placed on the precision conversion of the network layer weights. According to the computing power and memory resources of the embedded device, a suitable precision representation method is selected, such as converting 32-bit floating-point numbers to 16-bit or 8-bit fixed-point numbers, to reduce memory occupancy and computing load without significantly affecting model performance. At the same time, memory access is optimized, and the storage structure and data reading method of the citrus detection network model are adjusted to ensure efficient access to model weights and intermediate data under the limited memory bandwidth of the embedded device. For example, techniques such as cache strategy and data prefetching are used to reduce memory latency and improve data processing speed. In addition, the multi-core processor architecture of the embedded device is fully utilized to perform parallel computing on the model's computing process. By reasonably dividing the computing tasks and distributing the parallelizable operations to different cores for simultaneous execution, the inference speed of the citrus detection network model is further improved, enabling the citrus detection network model to quickly respond to users' detection requests on the embedded device.

[0126] After starting the detection system on the embedded device, the camera begins to collect image data within the recognition area. The camera should have appropriate resolution and frame rate to meet the real-time detection requirements, while ensuring that the image quality can clearly present the characteristics of the citrus. The collected image data is transmitted to the processing unit of the embedded end in a timely manner through the pre-set communication interface. At the embedded end, after receiving the image data, the optimized and converted citrus detection network model is immediately called for forward inference. The citrus detection network model generates prediction results for citrus, including category, location, and confidence information, through a series of convolution, pooling, and activation operations based on the feature information of the input image.

[0127] Finally, the recognition results are processed and displayed. For images in the video stream, the detection results can be superimposed into the video stream in the form of labeled boxes and text information, which is directly presented to the user. At the same time, the detection results can be saved in the local storage device at any time, and stored according to time, location, detection batch, etc. for easy user access to historical detection data. In addition, in order to realize remote management and sharing of data, the saved results can be uploaded to the cloud server through the network communication module, and presented in the form of visual charts and reports on the client terminal, so that relevant personnel can remotely monitor the detection of citrus, providing data support for the planting management and picking decision of citrus.

[0128] According to the experimental results in Table 1 below, the citrus detection network model has an average detection accuracy of 1.1% higher than the baseline model YOLOv7-Tiny, a total parameter reduction of 21.2%, a model storage space compression to 8.4MB, and an FPS increase to 64.7.

[0129] Table 1 improved model and benchmark model comparison

[0130]

[0131] As can be seen from Table 1, the improved scheme of the present application improves the precision and detection speed of the model on the basis of model lightweight. Figure 7 For the algorithm detection effect comparison chart of the present application under different light intensities, the upper part of the chart is the algorithm effect chart of the deep learning benchmark model YOLOv7-Tiny, and the lower part is the algorithm effect chart of the improved YOLOv7-Tiny orange detection model. The improved YOLOv7-Tiny orange detection model has better detection effect than the benchmark model, can detect some small fruit missed targets, and can detect the same fruit. The confidence of the orange detection model is generally better than that of the benchmark model; from Figure 8 It can be seen that the orange detection method based on improved deep learning in complex background proposed by the present application has far fewer repeated detections in complex environment than the original algorithm; from Figure 9 It can be seen that the orange detection method based on improved deep learning in complex background proposed by the present application also has good detection results on unripe orange fruits. In summary, the orange detection method based on improved deep learning in complex background proposed by the present application can achieve a good balance between reducing model parameter quantity and improving model expressiveness, which is conducive to the deployment of the model on edge devices.

[0132] The first embodiment of the present application provides a complex background citrus detection method based on improved deep learning, which comprises the following steps: collecting complex background citrus images in an orchard to obtain an initial data set; preprocessing the initial data set to obtain a first data set; labeling and format converting the first data set to obtain a target data set; taking a YOLOv7-tiny network model as a deep learning benchmark model to construct an improved YOLOv7-tiny network model as a citrus detection network model; training and evaluating the citrus detection network model; obtaining the optimal weight of the citrus detection network model and testing the citrus detection network model; obtaining the test result through the citrus detection model and calculating the precision and recall based on the intersection over union (IOU) to judge the accuracy of the detection. The citrus detection model of the complex background citrus detection method based on improved deep learning replaces the Conv module in the backbone network with the PConvBS module, reduces the redundant feature extraction and the number of model parameters, optimizes the memory usage and the amount of calculation, improves the feature extraction efficiency, replaces the ELAN module with the Faster-PELAN module to enhance the multi-scale feature aggregation capability, realizes efficient feature sampling and aggregation, uses the ELAN-H module in the neck network for multi-scale feature fusion to reduce the amount of calculation and the number of parameters, improve the inference speed and be suitable for real-time application, optimizes the backbone network and the detection head, reduces the number of down-sampling and removes the large target detection head to reduce the number of parameters, improve the inference speed and effectively detect small and medium-sized citrus targets, uses the alphaShape-IoU Loss as the loss function to solve the problem of inaccurate positioning of small fruit targets, converts the format of the model weight file to reduce the memory occupation and the amount of calculation, optimizes the memory access, uses the cache and data prefetching technologies to improve the data processing speed, the camera collects the image and transmits it to the embedded end, calls the optimized model for forward inference to generate the citrus prediction result, the detection result is displayed to the user in an intuitive form, can be saved locally and uploaded to the cloud server, is presented in a visual form, facilitates remote monitoring and provides data support for citrus management and picking decision-making. The improved citrus detection network model has higher average detection accuracy, fewer total parameters, smaller model storage space and higher FPS than the benchmark model, improves the accuracy and detection speed on the basis of lightweight model, has good detection effect in complex environment, can balance between reducing the number of parameters and improving the expressiveness, and is beneficial to deployment on edge devices.

[0133] The division of steps of the above methods is only for the purpose of clear description, and in implementation, one step can be combined or some steps can be split and decomposed into multiple steps, as long as the same logical relationship is included, which is within the protection scope of the present patent; adding insignificant modifications or introducing insignificant designs in the algorithm or process, but not changing the core design of the algorithm and process, are within the protection scope of the present patent.

[0134] The second embodiment:

[0135] As Figure 10 shown, the second embodiment of the present application provides a citrus detection system based on improved deep learning in a complex background, which comprises a dataset acquisition module 201, a dataset acquisition module 201, a dataset format conversion module 203, an improved modeling module 204, a model training and evaluation module 205, a model testing module 206, a detection result generation module 207.

[0136] Specifically, the dataset acquisition module 201 is used to acquire complex background citrus images in an orchard to obtain an initial dataset; the dataset preprocessing module 202 is used to preprocess the initial dataset to obtain a first dataset; the dataset format conversion module 203 is used to label and format convert the first dataset to obtain a target dataset; the improved modeling module 204 is used to construct an improved YOLOv7-tiny network model as a citrus detection network model with the YOLOv7-tiny network model as a deep learning benchmark model; wherein the improved YOLOv7-tiny network model includes replacing the Conv module in all CBS modules in the backbone network with the PConvBS module; replacing the ELAN module in the backbone network with the Faster-PELAN module in a feature fusion manner; using the ELAN-H module in the neck network for multi-scale feature fusion; optimizing the backbone network and the detection head, including reducing the downsampling times of the backbone network and removing the corresponding large target detection head; using the aShape-IoU Loss as the loss function of the detection head; the model training and evaluation module 205 is used to train and evaluate the citrus detection network model; the model testing module 206 is used to obtain the optimal weight of the citrus detection network model and test the citrus detection network model; the detection result generation module 207 is used to obtain the test result through the citrus detection model and calculate the precision and recall based on the intersection over union IOU to further judge the accuracy of the detection.

[0137] It is not difficult to find that the present embodiment is a system embodiment corresponding to the first embodiment, and the present embodiment can be implemented in cooperation with the first embodiment. The related technical details mentioned in the first embodiment are still valid in the present embodiment. In order to reduce repetition, they will not be described here. Accordingly, the related technical details mentioned in the present embodiment can also be applied in the first embodiment.

[0138] It is worth mentioning that each module involved in the embodiment is a logical module. In actual application, one logical unit can be one physical unit, or a part of one physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of the present application, units not closely related to solving the technical problems proposed in the present application are not introduced in the embodiment, but this does not mean that there are no other units in the embodiment.

[0139] The third embodiment of the present application relates to a network side server, as shown in the formula (1), comprising at least one processor 302;And the memory 301 connected with at least one processor 302;Wherein, the memory 301 stores instructions executable by at least one processor 302, and the instructions are executed by at least one processor 302 to enable at least one processor 302 to execute the above-mentioned data processing method. Figure 11

[0140] Wherein, the memory 301 and the processor 302 are connected in a bus mode, the bus can include any number of interconnected buses and bridges, and the bus connects one or more processors 302 and various circuits of the memory 301 together. The bus can also connect various other circuits such as peripheral devices, voltage regulators and power management circuits together, which are well known in the art, therefore, they will not be further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, which provide units for communicating with various other devices on the transmission medium. The data processed by the processor 302 is transmitted on the wireless medium through the antenna, further, the antenna also receives data and transmits the data to the processor 302.

[0141] The processor 302 is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interface, voltage regulation, power management and other control functions. And the memory 301 can be used to store the data used by the processor 302 in the execution operation.

[0142] The fourth embodiment of the present application relates to a computer readable storage medium, which stores a computer program. The computer program is executed by the processor to realize the citrus detection method in the first embodiment based on improved deep learning in complex background.

[0143] ​That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment methods can be completed by programs instructing the relevant hardware, the programs are stored in a storage medium, and the programs include a plurality of instructions for causing a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0144] The above-mentioned is only the embodiment of the present application, and the common knowledge of specific structures and characteristics in the scheme is not described in detail here. Those skilled in the art know all the common technical knowledge in the field of the present application before the filing date or the priority date, can know all the prior art in this field, and have the ability to apply conventional experimental means before this date. Those skilled in the art can improve and implement the present scheme based on their own ability under the guidance of the present application, and some typical known structures or known methods should not be an obstacle for those skilled in the art to implement the present application. It should be pointed out that, for those skilled in the art, without departing from the structure of the present application, a number of modifications and improvements can be made, which should be considered as the protection scope of the present application, and these will not affect the effect and practicality of the present application. The protection scope of the present application should be subject to the content of its claims, and the specific implementation mode and the like in the specification can be used to explain the content of the claims.

[0145] The above-mentioned is only the preferred embodiment of the present application, and does not limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for detecting citrus in a complex background based on improved deep learning, characterized in that, The method comprises the following steps: S1, collecting complex background citrus images in an orchard to obtain an initial data set; S2, preprocessing the initial data set to obtain a first data set; S3, labeling and format converting the first data set to obtain a target data set; S4, taking a YOLOv7-tiny network model as a deep learning benchmark model to construct an improved YOLOv7-tiny network model as a citrus detection network model; wherein the improved YOLOv7-tiny network model comprises replacing the Conv module in all CBS modules in the backbone network with a PConvBS module; replacing the ELAN module in the backbone network with a Faster-PELAN module in a feature fusion manner; adopting an ELAN-H module in the neck network for multi-scale feature fusion; optimizing the backbone network and the detection head, including reducing the number of down-sampling times of the backbone network and removing the corresponding large target detection head; and adopting an alphaShape-IoU Loss as a loss function for the detection head; The Faster-PELAN module is composed of a PConvBS module, a DWConvBS module, a Concat module and a CBS module; the Faster-PELAN module has two paths, the first path passes through the PConvBS module and the DWConvBS module for fast sampling, and the second path samples through the PConvBS module; the feature maps sampled on the specific layer on the first path are spliced with the feature maps sampled on the second path, and finally a convolution layer is used to complete the feature aggregation process; the PConvBS module is sequentially composed of a PConv module, a BatchNormalization layer and a SiLU activation function, and the DWConvBS module is sequentially composed of a DWConv module, a BatchNormalization layer and a SiLU activation function; The calculation formula of the loss function alphaShape-IoU Loss is: L αShape-IoU = a (1 - IoU) + (1 - a) distance shape + 0.5 x Ω shape wherein L αShape-IoU is a loss function, a is a weight parameter, IoU is an intersection over union, distance shape is a shape distance, and Ω shape is a shape complexity related term. S5, training and evaluating the citrus detection network model; S6, obtaining the optimal weight of the citrus detection network model and testing the citrus detection network model; S7, obtaining the test result through the citrus detection model and calculating the precision and recall based on the intersection over union (IOU) to judge the accuracy of the detection.

2. The improved deep learning based citrus detection method in complex background according to claim 1, wherein, In S2, the preprocessing of the initial data set comprises filtering, data enhancement and resolution formatting of the initial data set.

3. The method for citrus detection in complex background based on improved deep learning according to claim 2, characterized in that, In S2, the preprocessing of the initial data set to obtain the first data set comprises the following steps: Step S21, filtering the initial data set, including deleting images that are unclear and deleting citrus image data that lack key features; Step S22, data enhancement includes color space transformation and direction reversal of images; Step S23, resolution formatting includes resolution formatting processing of the initial data set, and a suitable standard resolution is selected according to the requirements of model training and the actual situation of hardware resources.

4. The method for citrus detection in complex background based on improved deep learning according to claim 1, characterized in that, In S3, the specific content of labeling the first data set includes labeling categories, labeling center points, and labeling width and height information, and the format conversion of the first data set includes reorganizing and extracting the labeling information in the xml file to obtain a txt format label file as the target data set.

5. The method for citrus detection in complex background based on improved deep learning according to claim 4, characterized in that, The formula for calculating the number of floating point operations and the memory access amount of the PConvBS module is as follows: where h is the height of the feature map, w is the width of the feature map, k is the size of the convolution kernel, c p is the number of output channels, where c p / c = 1 / 4.

6. The improved deep learning based citrus detection method in complex background according to claim 1, characterized in that, The formula for calculating the intersection over union IoU is as follows: wherein IoU is intersection over union, B is a predicted bounding box, B gt is a true bounding box; IoU is used to measure the overlap degree of the predicted box and the real box, and the value of IoU is between 0 and 1, and the larger the value is, the higher the coincidence degree of the predicted box and the real box is. Shape distance distance shape The formula for calculating the shape distance distance is: wherein distance shape is the shape distance, x c and y c are the coordinates of the center of the predicted box, and are the coordinates of the center of the real box, c is a constant, HH and WW are coefficients related to the width and height of the box, w gt is the width of the real box, h gt is the height of the real box, (w gt ) scale is a new width value obtained by scaling the width of the real box, (h gt ) scale is a new height value obtained by scaling the height of the real box; a shape complexity related term Ω shape The calculation formula is: where Ω shape is a shape complexity related term, w t is the shape of the box, and θ is a fixed parameter.

7. A complex background based orange detection system using improved deep learning, characterized by, The system is applied to the complex background citrus detection method based on improved deep learning in claim 1, and the system comprises: A data set acquisition module acquires complex background citrus images in an orchard to obtain an initial data set; A data set preprocessing module is configured to preprocess the initial data set to obtain a first data set; A data set format conversion module is configured to label and convert the format of the first data set to obtain a target data set; An improved modeling module is configured to use a YOLOv7-tiny network model as a deep learning benchmark model to construct an improved YOLOv7-tiny network model as a citrus detection network model; wherein the improved YOLOv7-tiny network model comprises replacing the Conv module in all CBS modules in the backbone network with the PConvBS module; replacing the ELAN module in the backbone network with the Faster-PELAN module by using a feature fusion method; using the ELAN-H module in the neck network for multi-scale feature fusion; optimizing the backbone network and the detection head, including reducing the number of down-sampling times of the backbone network and removing the corresponding large target detection head; and using the alphaShape-IoU Loss as the loss function of the detection head; A model training and evaluation module is configured to train and evaluate the citrus detection network model; A model testing module is configured to obtain the optimal weight of the citrus detection network model and test the citrus detection network model; A detection result generation module is configured to obtain a test result through the citrus detection model and calculate the precision and recall based on the intersection over union IoU to determine the accuracy of the detection.

8. A network-side server, characterized by, comprise: at least one processor; and a memory in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the complex background citrus detection method based on improved deep learning according to any one of claims 1 to 6.

9. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the complex background citrus detection method based on improved deep learning according to any one of claims 1 to 6.

Images