A fruit detection method and system under occlusion environment

Abstract

The present disclosure belongs to the technical field of agricultural machinery, and provides a fruit detection method and system under sheltered environment, comprising the following steps: acquiring an image of a fruit to be detected; extracting and fusing image features of the acquired image to obtain fused image features; and performing fruit detection according to the obtained fused image features and a shelter detection model; wherein the shelter detection model takes minimization of an image loss function as an objective function, and adopts a double-layer graph convolution network to realize fruit detection. The present disclosure improves detection speed under the premise of ensuring detection accuracy, solves the problem of mutual contradiction between accuracy and speed in previous fruit detection and segmentation methods, and meets actual operation requirements in actual orchard production.

Description

TECHNICAL FIELD

[0001] The present disclosure belongs to the technical field of agricultural machinery, and particularly relates to a fruit detection method and system in a sheltered environment. BACKGROUND

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute the prior art.

[0003] With the gradual maturity of deep learning technology, it has become a common phenomenon to transplant this new revolution into various industries to obtain better results, which has also stimulated the development of automatic robots in the agricultural field. As the most basic and important part of agricultural robots, the visual system is used to analyze the specified target from complex and diverse scenes and has been widely used in many practical applications, such as fruit yield estimation, crop growth monitoring, disease detection, etc. The accuracy, efficiency and robustness of the fruit detection of the visual recognition system as an important part of the fruit and vegetable picking robot will greatly affect the picking quality of the picking robot.

[0004] However, in a complex orchard environment, the visual system is easily disturbed by many factors such as light intensity and angle, leaf shelter, fruit color, etc., bringing great challenges to the effective detection of target fruits.

[0005] According to the inventors' understanding, fruit detection generally adopts two directions of machine learning and deep learning: among them, the method based on machine learning often accompanies a series of complex operations such as image preprocessing, feature selection and extraction, and the recognition effect of the model is also easily affected by these operations; in addition, this method detects target fruits based on the combination of other representations such as color, texture, shape, etc. of target fruits, when the texture features of the fruit surface are not obvious due to factors such as light intensity, angle or natural environment; the shape is missing due to factors such as branch shelter, leaf shelter or overlapping of fruits; the color interference caused by the target fruit and the background of the same color system of branches and leaves, etc. will all make the recognition accuracy of this method for target fruits decrease significantly. In the detection model based on deep learning, the accuracy and applicability are greatly improved compared with traditional visual methods, but this method requires a large amount of computing and storage resources, and the speed cannot meet the real-time needs of the picking robot, in addition, the power consumption and stability of the picking robot also need to be considered in actual orchard picking. SUMMARY

[0006] In order to solve the above problems, the present disclosure proposes a fruit detection method and system in a sheltered environment, which improves the detection speed under the premise of ensuring the detection accuracy, solves the problem of mutual contradiction between accuracy and speed in the previous fruit detection and segmentation method, and meets the actual operation needs in actual orchard production.

[0007] According to some embodiments, the first aspect of the present disclosure provides a fruit detection method in a shielding environment, which adopts the following technical solution:

[0008] A fruit detection method in a shielding environment, comprising the following steps:

[0009] Obtaining an image of a fruit to be detected;

[0010] Extracting and fusing image features of the obtained image to obtain fused image features;

[0011] Performing fruit detection according to the obtained fused image features and a shielding detection model;

[0012] The shielding detection model takes the minimization of an image loss function as an objective function, and realizes fruit detection by using a double-layer graph convolutional network.

[0013] As a further technical limitation, the obtained image of the fruit to be detected is a fruit image in various environments, including different time periods, different weather, different light angles, and different shielding conditions.

[0014] As a further technical limitation, in the process of image feature extraction, the image features are extracted based on an optimized ResNet.

[0015] As a further technical limitation, in the process of image feature fusion, a feature pyramid is used to integrate features through horizontal connection and top-down fusion between each feature map to obtain fused image features.

[0016] Further, after obtaining the fused image features, a non-local module is embedded based on the feature pyramid to perform fine processing of the image features.

[0017] As a further technical limitation, the shielding detection model processes the fused image features to distinguish the shielding part and the shielded part, and performs distribution processing on the distinguished image features; wherein the first layer graph convolutional network is used to detect the outline and process the shielded part, to realize outline prediction and mask regression of the shielded part; the second layer graph convolutional network is used to process the shielding part, to realize outline prediction and mask regression of the shielding part.

[0018] As a further technical limitation, the image loss function includes a detection stage loss sub-function, a shielded branch loss sub-function, and a shielding branch loss sub-function.

[0019] According to some embodiments, the second aspect of the present disclosure provides a fruit detection system in a shielding environment, which adopts the following technical solution:

[0020] A fruit detection system in a shielding environment, comprising:

[0021] An acquisition module configured to acquire an image of a fruit to be detected;

[0022] A fusion module configured to extract and fuse image features of the acquired image to obtain fused image features;

[0023] A detection module configured to perform fruit detection according to the obtained fused image features and a shielding detection model;

[0024] The shielding detection model takes minimization of an image loss function as an objective function, and adopts a double-layer graph convolution network to realize fruit detection.

[0025] According to some embodiments, a third aspect of the present disclosure provides a computer-readable storage medium, which adopts the following technical solution:

[0026] A computer-readable storage medium having a program stored thereon, the program being executed by a processor to implement the steps in the fruit detection method in a shielding environment according to the first aspect of the present disclosure.

[0027] According to some embodiments, a fourth aspect of the present disclosure provides an electronic device, which adopts the following technical solution:

[0028] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor, and the processor executes the program to implement the steps in the fruit detection method in a shielding environment according to the first aspect of the present disclosure.

[0029] Compared with the prior art, the present disclosure has the following beneficial effects:

[0030] The compression model capacity of the present disclosure reduces the calculation and storage resources, accelerates the training and detection speed of the model, and solves the contradiction between the precision and speed of the previous fruit detection method; improves the detection precision of the model under the interference of leaves, fruits, and other shielding, and meets the multiple requirements of speed, precision, and robustness in the actual application of various intelligent technologies. BRIEF DESCRIPTION OF DRAWINGS

[0031] The accompanying drawings, which form a part of the present disclosure, are used to provide a further understanding of the present disclosure, and the schematic embodiments of the present disclosure and their descriptions are used to explain the present disclosure, and do not constitute an improper limitation on the present disclosure.

[0032] Figure 1 is a flowchart of the fruit detection method in a shielding environment in the first embodiment of the present disclosure;

[0033] Figure 2 is a whole work flowchart of the fruit detection method in a shielding environment in the first embodiment of the present disclosure;

[0034] Fig. 3(a) is a real fruit image collected under different illumination angles in the first embodiment of the present disclosure;

[0035] Fig. 3(b) is a real fruit image collected under different occlusion types in the first embodiment of the present disclosure;

[0036] Fig. 3(c) is a real fruit image collected under different shooting distances in the first embodiment of the present disclosure;

[0037] Figure 4 is a detection flowchart of the whole in the first embodiment of the present disclosure;

[0038] Figure 5 is a non-local module schematic diagram of the fine feature pyramid embedding in the first embodiment of the present disclosure;

[0039] Figure 6 is an implementation detail diagram of the double-layer GCN in the first embodiment of the present disclosure;

[0040] Figure 7 is a detection effect diagram in the first embodiment of the present disclosure;

[0041] Figure 8 is a structural block diagram of the fruit detection system in the occlusion environment in the second embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] The present disclosure will be further described below in conjunction with the drawings and embodiments.

[0043] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs.

[0044] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should also be understood that when the terms "comprise" and / or "include" are used in the specification, there is a presence of the features, steps, operations, devices, components and / or combinations thereof.

[0045] The embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0046] Embodiment One

[0047] The first embodiment of the present disclosure introduces a fruit detection method in an occlusion environment.

[0048] AsFigure 1 The fruit detection method in the occluded environment shown includes the following steps:

[0049] An image of a fruit to be detected is acquired;

[0050] Image features of the acquired image are extracted and fused to obtain fused image features;

[0051] Fruit detection is performed according to the obtained fused image features and an occlusion detection model;

[0052] The occlusion detection model takes minimization of an image loss function as an objective function, and a double-layer graph convolution network is used to realize fruit detection.

[0053] This embodiment details the fruit detection method in the occluded environment, as shown in Figure 2 and Figure 4 The specific workflow is as follows:

[0054] Step S01: Image acquisition and data set preparation for fruit to be detected;

[0055] Step S02: Image feature extraction and fusion;

[0056] Step S03: Feature refinement processing;

[0057] Step S04: Occlusion region processing;

[0058] Step S05: Loss calculation and model training.

[0059] In step S01, to evaluate the detection effect of the fruit, this embodiment takes apples as an example for related research, collects and prepares an apple data set, the data set is shot by a Sony Alpha 7II camera, and the collected data set contains fruit images shot in various environments such as different time periods, different weather, different light angles, different occlusion conditions, etc.

[0060] Fig. 3(a), Fig. 3(b) and Fig. 3(c) respectively list real fruit images collected under different light angles, different occlusion types and different shooting distances; the target fruit in the image is manually labeled using the labelme software, before labeling, the image resolution is uniformly reduced to 600x400 pixels, the minimum circumscribed matrix of each fruit in the labeled image is taken as a real box, and the corresponding json file is generated and uniformly converted into the MS COCO data set format.

[0061] In step S02, the residual network ResNet50 is selected as the basic network to extract image features. The feature expression ability of deep feature maps is gradually enriched through convolution and pooling operations. Considering the distance between the robot and the object, as well as the small area of the occluded object, the vision system of the picking robot also needs to accurately identify small area fruits in the image. Therefore, a feature pyramid (Feature Pyramid Networks, FPN) is introduced into the model architecture. The features are integrated through horizontal connection and top-down fusion between each feature map to improve the final detection performance, and the features of each small area fruit are also extracted.

[0062] The last residual blocks conv2, conv3, conv4, and conv5 of ResNet50 output feature maps {A2, A3, A4, A5}. The feature maps are fused in a top-down and horizontal connection manner, so that the deep feature maps and shallow feature maps have the same semantic capacity, and the FPN {F2, F3, F4, F5} is obtained. By applying FPN to fruit recognition, the detection effect of the model on different scales, especially small scale target fruits, can be effectively improved, so as to better guide the identification and picking of the apple picking robot and path planning.

[0063] In step S03, in the ResNet+FPN structure, the comprehensive features pay more attention to the semantic information of adjacent pixels, and pay less attention to other resolutions. In the information flow process, the semantic information contained in the non-adjacent layers will be diluted at each fusion. Therefore, a non-local module is embedded in the FPN to obtain a refined feature pyramid, so as to obtain and refine more semantic feature information. The implementation details are shown in Figure 5 .

[0064] The result feature of each position is the weighted sum of all position features and the original feature. Therefore, it has a global semantic feature information, and selectively aggregates the semantic features according to the correlation strength matrix. Similar semantic features achieve mutual gain, thereby improving the semantic extraction of feature information, and also suppressing noise.

[0065] In step S04, the design of the occlusion part detection model is simple and effective, which is composed of a conv, a GCN layer and a FCN layer. Then the output is input to the upsampling layer and the convolution layer to obtain a channel feature map for joint boundary and mask prediction. The GCN is realized by a DANet module, which is divided into a position attention module and a channel attention module. The implementation details are shown in Figure 6 . The top GCN layer detects the occluded object, and the bottom GCN layer infers the partially occluded instance.

[0066] In the segmentation process using a two-layer GCN structure, the feature information obtained from the detection network is processed, especially for the discrimination of occluded objects, by processing the occluded part step by step. First, the occluded part and the occluded part are distinguished, and then sent to their respective GCN layers. Finally, the information is integrated to output the final predicted image. Specifically, the first GCN layer is used to detect contours and process occluded objects, achieving contour prediction and mask regression for occluded objects; the second GCN layer is used to process occluded objects, achieving contour prediction and mask regression for occluded objects. Based on the attention mechanism in GCN, it can focus more on useful information and reduce noise interference. After this two-layer GCN processing, the occluded object and the occluded object are processed separately and then integrated, achieving accurate detection of occluded objects.

[0067] In step S05, to achieve the ability to detect fruit, the loss between the predicted bounding box and the ground truth bounding box obtained by the model training is estimated through iterative training, and the model parameters are updated by backpropagating the gradient according to the loss value until the loss is reduced and converges to a value range, thus obtaining an optimal loss function for model training.

[0068] Specifically, based on the structural analysis of the model, the loss function L of this model should consist of three parts: the loss L generated during the detection phase. Detect Loss L of the shaded branch Occluder and the loss of the shading branch L Occludee The formula for the loss function L is shown below:

[0069] L = L Detect +L Occluder +L Occludee (1)

[0070] Regarding the loss L generated by the model during the detection phase Detect The loss is composed of three branches: Classification, Regression, and Centerness. Since the target fruit occupies a smaller area compared to the background in an image, and is further reduced by the σ factor, an imbalance exists between positive and negative samples during training. To address these drawbacks and simplify calculation, the Classification, Regression, and Centerness branches are calculated using FocalLoss, IoU (intersection of union) Loss, and BCE Loss, respectively. The overall loss function for the model's detection part is shown in the following equation:

[0071]

[0072] In the above formula, p x,y ,d x,y ,center x,y are the predicted values of the classification branch, the regression branch and the center degree branch at the spatial position (x, y) respectively, corresponding to the training target at the spatial position (x, y), among the three loss terms, L reg ,L center only for positive samples, N pos represents the number of positive samples, and λ, β are the balance coefficients of each loss term.

[0073] The loss functions generated by the segmentation network's occlusion branch and the occluded branch are L Occludee and L Occluder , whose function formulas are shown in the following formula:

[0074] L Occludee =λ1L Occ-B +λ2L Occ-S (3)

[0075] L Occluder =λ3L' Occ-B +λ4L' Occ-S (4)

[0076] Among them, L Occ-B and L Occ-S represent the boundary detection and mask segmentation loss of the occluded in the second layer GCN layer.

[0077] Regarding the classification loss L' Occ-B of the boundary detection of the detection occluder in the above formula, it is shown as follows:

[0078] L' Occ-B =L BCE (W B F occ (X roi ),gT B ) (5)

[0079] Among them, L BCE represents the binary cross-entropy loss, which is shown as follows:

[0080]

[0081] f occ represents the nonlinear transformation function of the occlusion modeling module; W B is the weight of the boundary predictor; X roi is the clipping FPN feature map given by the RoIAlign operation of the target region; gT B is the boundary of the ready-made occluder, which can be easily calculated from the mask annotation.

[0082] Regarding the classification loss L' in the above formula for modeling the occluder detection, Occ-S As shown in the following formula:

[0083] L' Occ-s =L BCE (W S F occ (X roi ),gT S (7)

[0084] Among them, F occ (X roi ) is the shared feature used in the joint optimization of boundary prediction in occluder mask prediction; W S GT represents the trainable weights of the segmentation mask predictions from a 1×1 convolutional layer. S This indicates the mask label for the blocker.

[0085] This embodiment uses an anchor-free, fully convolutional approach to compress model size, reduce computational and storage resources, and accelerate model training and detection speed without compromising the detection and segmentation accuracy of anchor-based methods. This resolves the contradiction between accuracy and speed in previous fruit detection methods; it also improves the model's detection accuracy under interference from leaves, fruit occlusion, etc., and meets the multiple requirements of various intelligent technologies for speed, accuracy, and robustness in practical applications.

[0086] Example 2

[0087] Embodiment 2 of this disclosure introduces a fruit detection system under shading conditions.

[0088] like Figure 8 The fruit detection system shown includes:

[0089] The acquisition module is configured to acquire an image of the fruit to be detected.

[0090] The fusion module is configured to extract and fuse image features from the acquired image to obtain fused image features.

[0091] The detection module is configured to detect fruit based on the obtained fused image features and the occlusion detection model;

[0092] The occlusion detection model uses the minimum image loss function as its objective function and employs a two-layer graph convolutional network to detect fruit.

[0093] The detailed steps are the same as the fruit detection method under shading conditions provided in Example 1, and will not be repeated here.

[0094] Example 3

[0095] Embodiment three of the present disclosure provides a computer readable storage medium.

[0096] A computer readable storage medium, having a program stored thereon, which, when executed by a processor, implements the steps in the fruit detection method in a shielding environment as described in embodiment one of the present disclosure.

[0097] The detailed steps are the same as the fruit detection method in a shielding environment provided in embodiment one, and will not be repeated here.

[0098] Embodiment four

[0099] Embodiment four of the present disclosure provides an electronic device.

[0100] An electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor implements the steps in the fruit detection method in a shielding environment as described in embodiment one of the present disclosure when executing the program.

[0101] The detailed steps are the same as the fruit detection method in a shielding environment provided in embodiment one, and will not be repeated here.

[0102] Although the specific embodiments of the present disclosure are described above with reference to the drawings, the present disclosure is not limited to the above-described embodiments, and various modifications or changes can be made to the embodiments without departing from the scope of the present disclosure.

Claims

1. A method for fruit detection under shading conditions, characterized in that, The method comprises the following steps: obtaining an image of a fruit to be detected; extracting and fusing image features of the obtained image to obtain fused image features; detecting the fruit according to the obtained fused image features and a shielding detection model; wherein the shielding detection model takes a minimum image loss function as an objective function and adopts a double-layer graph convolution network to realize fruit detection; in the process of image feature fusion, a feature pyramid is adopted, feature integration is performed through horizontal connection and top-down fusion between each feature map, and fused image features are obtained; after the fused image features are obtained, a non-local module is embedded on the basis of the feature pyramid to perform fine processing of the image features; the shielding detection model processes the fused image features, distinguishes a shielding part and a shielded part, and performs distribution processing on the distinguished image features; wherein a first layer of graph convolution network is used to detect an outline and process the shielded part, to realize outline prediction and mask regression of the shielded part; a second layer of graph convolution network is used to process the shielding part, to realize outline prediction and mask regression of the shielding part; the image loss function comprises a detection stage loss sub-function, a shielded branch loss sub-function, and a shielding branch loss sub-function.

2. A method of detecting fruit in an obscured environment as claimed in claim 1, wherein, The obtained image of the fruit to be detected is a fruit image in various environments, wherein the various environments include different time periods, different weathers, different light angles, and different shielding conditions.

3. A method for detecting fruits in an obstructed environment as claimed in claim 1, wherein, In the process of image feature extraction, image features are extracted based on an optimized ResNet.

4. A fruit detection system in a sheltered environment, using a fruit detection method in a sheltered environment according to any one of claims 1 to 3, characterized in that, The method comprises the following steps: an obtaining module configured to obtain an image of a fruit to be detected; a fusing module configured to extract and fuse image features of the obtained image to obtain fused image features; a detecting module configured to detect the fruit according to the obtained fused image features and a shielding detection model; wherein the shielding detection model takes a minimum image loss function as an objective function and adopts a double-layer graph convolution network to realize fruit detection.

5. A computer-readable storage medium having stored thereon a program, characterized in that, The program is executed by the processor to realize the steps in the fruit detection method in a shielding environment according to any one of claims 1-3.

6. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized by The processor executes the program to realize the steps in the fruit detection method in a shielding environment according to any one of claims 1-3.

Images