A visual recognition method for growth posture of young fruit in orchard based on multiple feature fusion

Abstract

The present invention relates to the field of image detection technology, and in particular to a method for visually identifying the growth posture of young fruit in an orchard based on multiple feature fusion. The method comprises collecting data images of young fruit in an orchard and adjusting a target detection frame in the image; performing format conversion on a labeled data set and performing cropping preprocessing on the converted data set; constructing a young fruit growth posture feature extraction model and using a Bi-FPN network to deeply fuse shallow feature maps and high-level feature maps of the feature extraction model; using a posture prediction layer to perform posture frame regression on the fused feature map and extract the target area; training the model using a training data set, saving the coordinates of the posture frame, and calculating the growth posture angle of the young fruit. The present invention provides an effective solution for achieving the mechanization, automation, and intelligence of intelligent bagging, ensuring timely and efficient bagging of young fruit, and reducing bagging operation costs.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method for visually identifying the growth posture of young fruit in an orchard based on multiple feature fusion. Background Art

[0002] Bagging is an important technology for producing green, high-quality, and high-end fruits and vegetables. Bagging can effectively reduce bird and insect damage to fruits and vegetables, prevent pesticide pollution, sunburn, wind and rain damage, scratches and deformation, and improve the color of fruits and vegetables. Bagging is also an indispensable part of the high-quality fruit planting and production process. However, bagging young fruits, like picking ripe fruits, is a seasonal and labor-intensive task. Currently, it is mainly completed by hand or by hand with simple machinery. It is time-consuming, labor-intensive, and the quality of bagging varies. In addition, the aging and insufficient agricultural labor force is becoming increasingly prominent, and the labor cost of manual bagging is increasing year by year, which also increases the corresponding production costs, thereby affecting its market competitiveness.

[0003] When bagging young fruits, since the bags must be put on from the bottom of the fruit upwards, information on their growth posture is essential. Compared with mature fruits, young fruits are not only small in size but also have a color similar to the background of branches and leaves. Therefore, the difficulty of identifying their growth posture can be imagined. Summary of the Invention

[0004] Aiming at the deficiencies of existing algorithms, the present invention provides an effective solution in realizing the mechanization, automation and intelligence of intelligent bagging, ensuring the timely and efficient bagging of young fruits, and reducing the cost of bagging operations.

[0005] The technical solution adopted by the present invention is: a method for visually identifying the growth posture of young fruit in an orchard based on multiple feature fusion comprises the following steps:

[0006] Step 1: Collect orchard young fruit data images, adjust the target detection frame in the image, and enhance and annotate the image;

[0007] Furthermore, the target detection frame is adjusted by replacing the horizontal frame with a posture frame with angle parameters through rolabelImg, where the posture frame includes the target center point coordinates, length, width, and tilt angle.

[0008] Step 2: Convert the format of the labeled dataset and perform cropping and preprocessing on the converted dataset;

[0009] Furthermore, the format conversion is to express the tilt angle of the fruit according to the angle formed by the long side of the posture frame and the x-axis clockwise.

[0010] Step 3: Build a young fruit growth posture feature extraction model, and use the Bi-FPN network to deeply fuse the shallow feature map and high-level feature map of the feature extraction model;

[0011] Furthermore, the feature extraction model includes a Focus module, a feature extraction module P1, a CBL module, a feature extraction module P2, a CBL module, a feature extraction module P3, a CBL module, a feature extraction module P4, a CBL module and a feature extraction module P5, which are connected in sequence; among them, the feature extraction module P2 is composed of 2 BottleneckCSP modules + a CA attention mechanism module, the feature extraction modules P3 and P4 are composed of 8 BottleneckCSP modules + a CA attention mechanism module, and the feature extraction module P5 is composed of an SPP module, 2 BottleneckCSP modules + a CA attention mechanism module.

[0012] The Focus module uses a 2×2 frame with a step size of 2 to take pixel values ​​at intervals of one pixel in the image. The values ​​at fixed positions in each frame are placed in the same layer to obtain four images, which are then spliced ​​into a new image. The new image is convolved to obtain a sampling feature map.

[0013] The CBL module consists of convolution + BatchNormalization normalization + LeakReLU activation function.

[0014] The SPP module sends the sampled feature map to three pooling layers with different convolution kernel sizes of 1×1, 2×2, and 4×4;

[0015] The BottleneckCSP module concats two branches and performs BatchNormalization and a LeakReLU activation function. One branch consists of two CBL layers and a convolutional layer, and the other consists of a convolutional layer.

[0016] Furthermore, the Coordinate information embedding of the CA attention mechanism module is used to globally pool the input feature map in both horizontal and vertical directions to obtain feature maps in both horizontal and vertical directions;

[0017] In the Coordinate Attention generation transformation, the two feature maps are spliced ​​together, the feature map F1 is obtained through convolution transformation, and the feature map f is normalized;

[0018] Decompose the feature map f horizontally and vertically into f w ∈R C / r×W and f h ∈R C / r×H , r is the reduction rate, respectively for f wand f h Perform 1×1 convolution to obtain the feature map F w and F h , using the sigmoid activation function, we get the attention weights g of the feature map in two spatial directions respectively w and g h ;

[0019] Multiply the original feature map with the attention weights in the horizontal and vertical directions, and apply attention in both horizontal and vertical directions to the input features simultaneously.

[0020] Furthermore, the Bi-FPN network is used to deeply fuse the shallow feature map of the feature extraction model with the high-level feature map, including:

[0021] First, the P5 feature layer is upsampled and fused with the P4 feature layer; secondly, the obtained fused feature information is upsampled again and fused with the P3 feature layer for the second time; finally, the feature information obtained by the second fusion is upsampled and fused three times with the P2 feature layer to complete the top-down feature information fusion; similarly, the information obtained by the three fusions is first downsampled and fused with the feature information obtained by the second fusion; secondly, the fused information is downsampled and fused with the feature information obtained by the first fusion; finally, the fused information is downsampled and fused with the P5 feature map.

[0022] Step 4: Use the posture prediction layer to perform posture box regression on the fused feature map and extract the target area;

[0023] Furthermore, the head structure of the prediction layer adds an angle prediction channel. The channel dimension of the head detection layer is 3×(C+5+180), where 3 represents the anchor boxes with three aspect ratios preset for each grid. Each anchor box is responsible for predicting C categories and border parameter information (x, y, w, h, p r ); Category prediction channel (C0, C1, ..., C n ), p r Indicates the foreground confidence of the prediction box, and each anchor box will have an additional 180-channel angle prediction.

[0024] Step 5: Train the model using the training data set, save the coordinates of the posture frame, and calculate the growth posture angle of the young fruit;

[0025] Furthermore, the coordinates of the posture frame include the coordinates of the four vertices. The angle formed by the long side and the x-axis clockwise is the growth posture angle of the young fruit. The calculation formula is:

[0026]

[0027] Among them, (x1, y1), (x2, y2), (x3, y3) and (x4, y4) are the coordinates of the four vertices.

[0028] Beneficial effects of the present invention:

[0029] 1. Aiming at the problem of visual recognition of the growth posture of young fruits in an orchard environment, the position information of the young fruits is obtained as well as the posture angle information.

[0030] 2. The attention mechanism is introduced to solve the problem of close-color background, and the recognition performance is improved compared with not using the attention mechanism.

[0031] 3. In order to further improve the network's ability to recognize smaller targets, a small target detection layer is added, which can better identify the growth posture of young peach fruits. The missed detection rate is greatly reduced compared with not using the small target detection layer.

[0032] 4. Provide effective solutions in realizing the mechanization, automation and intelligence of intelligent bagging, ensuring timely and efficient bagging of young fruits, and reducing bagging operation costs, and also provide reference for solving the research problems of bagging for other fruits and vegetables. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a young fruit growth posture feature extraction model diagram of the present invention;

[0034] Figure 2 This is the structural model diagram of the CA attention mechanism of the present invention;

[0035] Figure 3 This is a Bi-FPN image with small target detection added to the present invention;

[0036] Figure 4 This is a visualization diagram of the growth posture angles of young peach fruits of the present invention. DETAILED DESCRIPTION

[0037] The present invention will be further described below in conjunction with the accompanying drawings and embodiments. This figure is a simplified schematic diagram, which only illustrates the basic structure of the present invention in a schematic manner, and therefore only shows the components related to the present invention.

[0038] A method for visually identifying the growth posture of young fruit in an orchard based on multiple feature fusion includes the following steps:

[0039] Step 1: Collect orchard young fruit data images, adjust the target detection frame in the image, and enhance and annotate the image;

[0040] By collecting young fruit datasets in different time periods and weather conditions, the dataset is made closer to different scenes under natural growth conditions, enhancing the adaptability of the network. Then, through image enhancement, the contrast enhancement factor is set to 1.5, the brightness is enhanced to 1.5, rotation is performed, and Gaussian noise is added to enrich the image data. Among them, the target detection frame is adjusted by replacing the horizontal frame with a posture frame with angle parameters through rolabelImg. The posture frame includes the coordinates of the target center point, length, width, and tilt angle, and obtains the posture information dataset.

[0041] Step 2: Convert the format of the labeled dataset and perform cropping and segmentation preprocessing on the converted dataset;

[0042] The format conversion is performed by converting the coordinates of the four vertices of the posture frame according to the angle formed by the long side of the posture frame and the x-axis clockwise to represent the tilt angle of the fruit.

[0043] Construct the training set and test set of the preprocessed young fruit dataset; the preprocessing includes: segmenting the young fruit images by cropping, and then using the image enhancement Mosaic data enhancement method. Mosaic enhances the data format and then divides the training set and validation set in a ratio of 8:2.

[0044] Step 3: Build a young fruit growth posture feature extraction model, and use the Bi-FPN network to deeply fuse the shallow feature map and high-level feature map of the feature extraction model;

[0045] like Figure 1 It is a feature extraction model consisting of Focus, CBL (Convolution-BatchNormalization-LeakReLU), Spatial Pyramid Pooling (SPP), Bottleneck CSP, and an attention mechanism module, which is used to extract image features.

[0046] Among them, the principle of the Focus module is to perform a series of slicing operations on the image before it enters the backbone network. The specific operation is to use a 2×2 frame with a step size of 2, take a value for every other pixel in each image, and put the value of each fixed position of the frame on the same layer to obtain four images, that is, to concentrate the W and H information into the channel space, and expand the input channel by 4 times; finally, the spliced ​​image becomes 12 channels compared to the RGB three-channel mode of the original image, and then the obtained new image is subjected to a convolution operation, and finally a double-downsampled feature map is obtained without information loss, which reduces the amount of calculation and speeds up the speed; among them.

[0047] CBL is composed of convolution + BatchNormalization + LeakReLU activation function, which can improve the effective information of extracting image features.

[0048] The SPP module sends the feature map of the previous layer to three pooling layers with different convolution kernel sizes of 1×1, 2×2, and 4×4, realizing the fusion of multiple receptive fields and improving the model's detection ability for complex scenes.

[0049] BottleneckCSP is composed of two branches concat, followed by BatchNormalization normalization and a LeakReLU activation function; one branch consists of two CBL layers and a convolutional layer, and the other consists of a convolutional layer. Its structure integrates the idea of ​​residual structure and connects two different branches to achieve feature fusion at different levels, greatly improving the network's feature extraction capability.

[0050] For small target detection processing, a prediction head for small target detection is added to the P2 layer (160×160 pixels) feature map of the backbone network to predict targets of 4×4 pixel size.

[0051] like Figure 2 As shown in Figure 3, the attention mechanism module focuses on important areas of the image and ignores irrelevant target areas, which better separates the target from the background.

[0052] The present invention uses the Coordinate information embedding in the CA attention mechanism module to perform global pooling on the input feature map in both horizontal and vertical directions. Specifically, given an input x, first use pooling with sizes (1, W) and (H, 1) to encode each channel along the horizontal coordinate and vertical coordinate respectively to obtain feature maps in both horizontal and vertical directions; then, in the Coordinate Attention generation transformation, the above two feature maps are spliced ​​together, and the feature map F1 is obtained by convolution transformation, which is normalized to obtain the feature map f; then the feature map f is decomposed into f along the horizontal and vertical directions. w ∈R C / r×W and f h ∈R C / r×H , r is the reduction rate, respectively for f w and f h Perform a 1×1 convolution calculation to obtain a feature map F with the same number of channels as the original one w and F h , using the sigmoid activation function, we get the attention weights g of the feature map in two spatial directions respectively w and g h; Finally, the original feature map is multiplied by the attention weights in the horizontal and vertical directions, and the attention in both horizontal and vertical directions is applied to the input features at the same time; CA attention obtains the relationship between channels and uses precise position information to capture long-term dependencies, which can better focus on target features to weaken background noise. The significance of the research is that young fruits belong to a close-colored background, which can improve the detection effect.

[0053] like Figure 3 Through the Bi-FPN network structure, the shallow feature maps and high-level feature maps of the feature extraction model are repeatedly fused from top to bottom and from bottom to top. Bi-FPN deeply fuses the P2-P5 feature layers. First, the P5 feature layer is upsampled and fused with the P4 feature layer. Second, the fused feature information is upsampled again and fused with the P3 feature layer for a second time. Finally, the feature information obtained from the second fusion is upsampled and fused three times with the P2 feature layer to complete the top-down feature information fusion. Similarly, the information obtained from the three fusions is downsampled and fused with the feature information obtained from the second fusion. Second, the fused information from the previous step is downsampled and fused with the feature information obtained from the first fusion. Finally, the fused information from the previous step is downsampled and fused with the P5 feature layer to complete the bottom-up feature information fusion, making full use of shallow information and reducing the negative impact of object scale.

[0054] Step 4: Use the posture prediction layer to perform posture box regression on the fused feature map to extract the target area;

[0055] Furthermore, the head structure of the posture prediction layer adds an angle parameter θ1. The angle parameter θ1 dimension contains 180 prediction angle channels (1, 2, 3...180), thereby converting the angle regression task into classification, so that the growth posture angle prediction can be achieved. Among them, the head detection layer channel dimension of the orchard young fruit growth posture visual recognition network is 3×(C+5+180), where 3 means that each grid will preset 3 types of anchor frames with different length and width ratios. Each anchor frame is responsible for predicting C categories and border parameter information (x, y, w, h, p r ); Category prediction channel (C0, C1, ..., C n ), p r Indicates the foreground confidence of the prediction box, and each anchor box will have an additional 180-channel angle prediction.

[0056] According to the growth characteristics of young fruits in the natural state of the orchard, their growth direction is mostly downward and has a certain angle, which brings great challenges to the research of intelligent bagging; to ensure that young fruits growing at multiple angles can be bagged, a posture frame is used to detect the target position, and the angle between the long side of the posture frame and the x-axis is used to represent the inclination angle of the fruit, so as to solve the problem of difficulty in finding the target and determining the bagging angle during the bagging process of young fruits; the background of young fruit images collected in an unstructured wild growth environment is complex and similar in color to the branches and leaves. It belongs to target detection under a close-color background, which makes it difficult to identify them. Therefore, the present invention replaces the last layer of the BottlenckCSP of the backbone with an attention mechanism, so that the attention is focused on important areas of the image and irrelevant target areas are ignored. It can not only capture position information and channel relationships at the same time, but also obtain long-range dependencies, enhance the feature representation of the network, and improve the recognition effect to a certain extent in a complex environment. The young fruits in the orchard are small, and their identification belongs to the problem of weak target detection in a complex background. The present invention uses the shallow feature map of the backbone to detect young fruit targets larger than 4×4, allowing the network to pay more attention to the detection of small targets and better learn the feature information of small targets, solving the problem of small target information loss caused by excessive downsampling multiples, and improving the target recognition effect.

[0057] Step 5: Train the orchard young fruit growth posture visual recognition network model using the training data set, use the obtained pre-trained weights to predict the test set, return the category name and confidence level, and finally save the coordinates of the posture box to calculate the growth posture angle.

[0058] like Figure 3 As shown in the figure, the posture angle is detected by the orchard young fruit growth posture visual recognition network based on multiple feature fusion, and the four vertex coordinates of the target will be obtained, namely ((x1, y1), (x2, y2), (x3, y3), (x4, y4)). The coordinates are counted clockwise from the upper left corner. In order to unify the detection direction of the posture target, this paper proposes that the angle formed by the long side and the x-axis clockwise is expressed as the growth posture angle of the young fruit. This method uses the prediction results to additionally calculate the angle of the target. According to the predicted coordinates, it is calculated by formula (1):

[0059]

[0060] Among them, by comparing the side lengths of the posture frame, the inverse trigonometric function arctan is used to calculate the angle of the slope corresponding to the longer side, which represents the growth posture angle θ of the young fruit.

[0061] Experimental results:

[0062] Taking young peach fruits in orchards as the research object, in order to explore the influence of different network models on the recognition of the growth posture angle of young fruits, the network designed in this experiment was compared with R3Det and R-CenterNet. The recognition results show that the visual recognition network of young fruit growth posture in orchards with multiple features fusion has the best network performance among the three models. This experiment verified the recognition effect of 300 test images by training weights. Figure 4 As shown in the figure, the experiments show that the average accuracy and angle estimation average accuracy of the model are the best, so the network can effectively identify the growth posture of young peach fruits.

[0063] With the above-described preferred embodiments of the present invention as a guide, and with reference to the above description, relevant personnel are fully capable of making various changes and modifications without departing from the technical scope of this invention. The technical scope of this invention is not limited to the contents of the specification and must be determined according to the scope of the claims.

Claims

1. A method for visually identifying the growth posture of young fruit in an orchard based on multiple feature fusion, characterized in that: The following steps are involved: Step 1: Collect orchard young fruit data images, adjust the target detection frame in the image, and enhance and annotate the image; Step 2: Convert the format of the labeled dataset and perform cropping and preprocessing on the converted dataset; Step 3: Build a young fruit growth posture feature extraction model, and use the Bi-FPN network to deeply fuse the shallow feature map and high-level feature map of the feature extraction model; The feature extraction model includes a Focus module, a first CBL module, a feature extraction module P2, a second CBL module, a feature extraction module P3, a third CBL module, a feature extraction module P4, a fourth CBL module, and a feature extraction module P5, which are connected in sequence. Among them, the feature extraction module P2 consists of two Bottleneck CSP modules and a CA attention mechanism module, the feature extraction modules P3 and P4 consist of eight Bottleneck CSP modules and a CA attention mechanism module, and the feature extraction module P5 consists of an SPP module, two Bottleneck CSP modules, and a CA attention mechanism module. The Coordinate information of the CA attention mechanism module is used to embed the input feature map in both horizontal and vertical directions to obtain feature maps in both horizontal and vertical directions. In the Coordinate Attention generation transformation, the two feature maps are spliced ​​together and the feature map is obtained through convolution transformation. F 1. Normalize to get the feature map f ; The feature map f Decomposed horizontally and vertically into f w ∈R C / r×W and f h ∈R C / r×H , r is the reduction rate, respectively f w and f h Perform 1×1 convolution to obtain the feature map F w and F h , using the sigmoid activation function, we can get the attention weights of the feature map in two spatial directions respectively g w and g h ; Multiply the original feature map with the attention weights in the horizontal and vertical directions, and apply attention in both horizontal and vertical directions to the input features simultaneously; Step 4: Use the posture prediction layer to regress the posture frame of the fused feature map and extract the target area; Step 5: Train the model using the training data set, save the coordinates of the posture frame, and calculate the growth posture angle of the young fruit.

2. The method for visually identifying the growth posture of young fruit in an orchard based on multiple feature fusion according to claim 1, characterized in that: The target detection frame is adjusted by replacing the horizontal frame with a posture frame with angle parameters through rolabelImg, where the posture frame includes the target center point coordinates, length, width, and tilt angle.

3. The method for visually identifying the growth posture of young fruit in an orchard based on multiple feature fusion according to claim 1, characterized in that: The format conversion is to express the tilt angle of the fruit according to the angle formed by the long side of the posture frame and the x-axis clockwise.

4. The method for visually identifying the growth posture of young fruit in an orchard based on multiple feature fusion according to claim 1, characterized in that: The Bi-FPN network is used to deeply fuse the shallow feature map and high-level feature map of the feature extraction model. Specifically, First, the P5 feature layer is upsampled and fused with the P4 feature layer; secondly, the obtained fused feature information is upsampled again and fused with the P3 feature layer for the second time; finally, the feature information obtained by the second fusion is upsampled and fused three times with the P2 feature layer to complete the top-down feature information fusion; similarly, the information obtained by the three fusions is first downsampled and fused with the feature information obtained by the second fusion; secondly, the fused information is downsampled and fused with the feature information obtained by the first fusion; finally, the fused information is downsampled and fused with the P5 feature map.

5. The method for visually identifying the growth posture of young fruit in an orchard based on multiple feature fusion according to claim 1, characterized in that: The head structure of the posture prediction layer adds an angle prediction channel. The channel dimension of the head detection layer is 3×(C+5+180), where 3 represents the anchor boxes with three aspect ratios preset for each grid. Each anchor box is responsible for predicting C categories and bounding box parameter information (x, y, w, h, p r ); Category prediction channel (C0, C1, ..., C C ), p r Indicates the foreground confidence of the prediction box, and each anchor box will have an additional 180-channel angle prediction.

6. The method for visually identifying the growth posture of young fruit in an orchard based on multiple feature fusion according to claim 1, characterized in that: The coordinates of the posture frame include the coordinates of the four vertices. The angle formed by the long side and the x-axis clockwise is the growth posture angle of the young fruit. The calculation formula is: (1) in,( x 1, y 1), ( x 2, y 2) ( x 3, y 3) are the coordinates of the three vertices.

Images