Lightweight semantic segmentation method for strawberry flower and fruit recognition in complex scene

Abstract

The invention relates to a lightweight semantic segmentation method for strawberry flower and fruit recognition in a complex scene, and the method comprises the following steps: obtaining a data set of strawberry images, and carrying out the image collection of strawberry flowers and fruits in a scene covering different illumination, time and plant states; carrying out data processing, and carrying out classification and mask labeling on the screened strawberry images; a deep and narrow backbone network is designed, and data features of rich and complex strawberry images are obtained; introducing more attention to the boundary of the strawberry image through an edge attention mechanism guiding model to obtain a clear edge; a pyramid pooling module is introduced at a de-wharf, and global information is perceived by aggregating strawberry image features of different scales; the strawberry image is segmented through the convolutional neural network, so that strawberry image segmentation in a complex environment is dealt with, and the lightweight and accuracy of the model are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of machine vision, and in particular to a lightweight semantic segmentation method for identifying strawberry flowers and fruits in complex scenes. Background Art

[0002] At present, there are two main forms of semantic segmentation methods for berry crops, namely machine learning and deep learning. Among them, the strawberry segmentation method based on deep learning can better solve the dilemma of artificially preset parameters. Talha et al. (2021) proposed a strawberry segmentation method based on encoder-decoder to identify three strawberries of different maturity, with an average intersection-over-union ratio of 88.79%, but did not consider the reality of cluttered background. Pérez-Borrero et al. (2020) proposed a lightweight strawberry segmentation method, and the inference time was increased from 5 frames per second (Frameper Second, FPS) to 10 frames per second, but only the size of strawberries could be classified. Subsequently, Li et al. (2021) proposed an apple segmentation method based on U-Net (Ronneberger et al., 2015). By designing an edge attention structure containing residual blocks and gated convolutions, the network can effectively extract the boundary information of green apples. Wang et al. (2023) used SwinTransformer as the backbone network for strawberry feature extraction, and the mIoU reached 90.15%, but the hardware processing power requirements were high. Recently, the general target segmentation method SAM (Kirillov et al., 2023; Mazurowski et al., 2023) has attracted widespread attention in the segmentation field. It can segment unlearned target types without additional training; however, the segmentation effect of SAM on agricultural targets such as strawberries is not satisfactory. This is because SAM can only generate the corresponding mask, but cannot know the target type under the mask, and cannot meet the requirements of real-time segmentation. In particular, the segmentation method based on machine learning requires specific lighting, shooting angles and scenes to obtain better segmentation results, and is extremely dependent on the parameters preset in advance by artificial experience, which is not applicable in the complex actual strawberry planting environment (Durand-Petiteville et al., 2019; Kumar et al., 2020).

[0003] In summary, the strawberry segmentation method based on deep learning is difficult to achieve a good balance between the lightweight and accuracy of the model, and cannot cope with segmentation tasks in complex environments. In addition, the existing strawberry segmentation method only recognizes and segments the strawberry fruit, but does not recognize and segment the strawberry flower at the same time. In order to achieve timely pollination and ripe fruit picking, it is necessary to recognize and segment both the strawberry flower and the fruit at the same time. Summary of the invention

[0004] The purpose of the present invention is to provide a lightweight semantic segmentation method for strawberry flower and fruit recognition in complex scenes, aiming to solve the problem of strawberry image segmentation in complex environments.

[0005] In order to solve the above technical problems, a lightweight semantic segmentation method for complex scene strawberry flower and fruit recognition is provided, which is characterized by comprising the following steps:

[0006] S1. Obtain a dataset of strawberry images, and collect images of strawberry flowers and fruits in scenes covering different lighting, time, and plant states;

[0007] S2, data processing, classifying and masking the screened strawberry images;

[0008] S3, designing a deep and narrow backbone network to obtain rich and complex data features of the strawberry image;

[0009] S4, guiding the model to pay more attention to the boundary of the strawberry image through the edge attention mechanism to obtain clear edges;

[0010] S5. A pyramid pooling module is introduced at the decoding head to perceive global information by aggregating the strawberry image features of different scales;

[0011] S6. Segment the strawberry image through a convolutional neural network.

[0012] Furthermore, the image collection of strawberry flowers and fruits in scenes covering different lighting, time and plant states includes:

[0013] The period from the appearance of the first flower to the ripening of the fruit of strawberry was selected as the data collection time range, which lasted for two months, with two days between each shooting;

[0014] Strawberry images were captured at four time periods: 8:00, 12:00, 16:00, and 20:00, and each capture lasted for one hour.

[0015] Furthermore, the capturing of strawberry images includes: capturing once each at three angles, namely, looking straight ahead, looking down, and looking up, from a distance of 50 cm from the strawberry growing rack, and capturing once at a straight ahead angle at a distance of 100 cm from the growing rack. During each capture, the robot is set to move 5 cm per second in the same direction, and 30 images are captured per second.

[0016] Furthermore, the mask annotation includes dividing the strawberry image into ripe fruits, unripe fruits, flowers and background.

[0017] Furthermore, after the mask annotation completes the annotation of the strawberry images, a strawberry data set is formed, and part of the strawberry images are randomly selected from the strawberry data set to form a training set, part of the strawberry images are formed to form a test set, and the remaining part of the strawberry images are formed to form a verification set.

[0018] Furthermore, the deep and narrow backbone network includes a larger number of convolution kernels and a smaller number of convolution channels.

[0019] Furthermore, the number of convolution kernels is 7×7, and a residual structure is introduced to connect the convolutions in pairs.

[0020] Furthermore, the edge attention mechanism includes activating a spatial information weight map through Sigmoid to generate a two-dimensional spatial edge attention map.

[0021] Furthermore, the strawberry image includes strawberry objects of different sizes, and the pyramid pooling module is used to eliminate the lack of global understanding ability of the model caused by the edge attention mechanism.

[0022] Furthermore, the network architecture of the strawberry image segmentation is LSSNet.

[0023] Implementing the embodiments of the present invention will have the following beneficial effects:

[0024] 1. The lightweight semantic segmentation method for complex scene strawberry flower and fruit recognition in this embodiment, because the strawberry picture data set not only masks and marks the ripe and to-be-ripened strawberries, but also newly adds pictures of strawberry flowers, so that the strawberry image contains a large number of difficult samples that reflect the actual strawberry planting scene, such as leaf occlusion, fruit overlap, backlighting, distant view, and jitter, so that the collected strawberry data is closer to the actual strawberry planting environment and robot operation scene, enriching the extensiveness of the strawberry data set;

[0025] 2. The lightweight semantic segmentation method for complex scene strawberry flower and fruit recognition in this embodiment proposes a deep and narrow backbone network architecture based on the exploration of the relationship between the depth and width of the network model, thereby achieving better segmentation and recognition performance while greatly reducing the number of parameters through a larger number of convolution kernels and a smaller number of convolution channels.

[0026] 3. The lightweight semantic segmentation method for complex scene strawberry flower and fruit recognition in this embodiment guides the model to pay more attention to the boundaries of the strawberry image through the edge attention mechanism to obtain clear edges, thereby reducing the problem of blurred edges of strawberries in the image caused by objective factors such as occlusion. In addition, the pyramid pooling module is introduced to enhance the model's recognition ability of strawberries of different sizes. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1 It is a flow chart of a lightweight semantic segmentation method for identifying strawberry flowers and fruits in complex scenes according to an embodiment of the present invention;

[0029] Figure 2 A schematic diagram of a robot image acquisition device according to an embodiment of the present invention;

[0030] Figure 3 The robot according to the embodiment of the present invention collects strawberry images from the growing rack;

[0031] Figure 4 The handheld camera described in the embodiment of the present invention collects strawberry images;

[0032] Figure 5 Schematic diagram of a strawberry dataset according to an embodiment of the present invention, wherein (a)-(h) are difficult samples reflecting actual strawberry planting scenes, and (g)-(k) are strawberry images from the public dataset SS1K;

[0033] Figure 6 Schematic diagram of the edge attention module according to an embodiment of the present invention, wherein stride represents the step size of the convolution kernel movement, C represents the number of input channels, and C' represents the number of output channels;

[0034] Figure 7 Schematic diagram of a pyramid pooling module according to an embodiment of the present invention;

[0035] Figure 8 Schematic diagram of the overall network architecture of LSSNet described in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. The preferred embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0037] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0038] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0039] Please refer to Figure 1-Figure 8 The embodiment of the present invention provides a lightweight semantic segmentation method for complex scene strawberry flower and fruit recognition, characterized in that it includes the following steps:

[0040] S1. Obtain a dataset of strawberry images, and collect images of strawberry flowers and fruits in scenes covering different lighting, time, and plant states;

[0041] S2, data processing, classification and mask annotation of the screened strawberry images;

[0042] S3, design a deep and narrow backbone network to obtain rich and complex data features of strawberry images;

[0043] S4, guide the model to pay more attention to the border of the strawberry image through the edge attention mechanism to obtain clear edges;

[0044] S5, a pyramid pooling module is introduced at the decoding head to perceive global information by aggregating strawberry image features of different scales;

[0045] S6, segmenting strawberry images by convolutional neural network. In specific applications, in order to solve the problem that the segmentation accuracy of existing neural networks is not high in complex strawberry planting environments, the present invention forms a strawberry image data set by collecting strawberry images that can reflect the actual strawberry planting scene, and constructs the segmentation and recognition of strawberry flowers and fruits in complex production scenes based on this data set. The present invention selects U-Net as the baseline model for the segmentation of strawberry flowers and fruits. However, U-Net still has many limitations, such as large model parameter amount and not high segmentation accuracy, which is difficult to meet the needs of real-time operation of mobile devices. For this reason, the present invention further improves U-Net to make it a lightweight model that can realize high-precision segmentation and recognition of strawberries in complex strawberry planting scenes.

[0046] At present, semantic segmentation methods based on deep learning have not been able to achieve simultaneous recognition and segmentation of flowers and fruits with lightweight deep models for strawberries in complex actual planting scenes, such as the background and the target are close, and the flowers and fruits occlude each other. The present invention improves U-Net to make it a lightweight model that can achieve high-precision segmentation and recognition of strawberries in complex strawberry planting scenes. Compared with the baseline model U-Net, the mIoU of the LSSNet proposed in the present invention is improved by 6.10% and mAcc is improved by 8.56%.

[0047] In a possible implementation, capturing images of strawberry flowers and fruits in scenes covering different lighting, time, and plant states includes:

[0048] The period from the appearance of the first flower to the ripening of the fruit of strawberry was selected as the data collection time range, which lasted for two months, with two days between each shooting;

[0049] Strawberry images were taken at 8:00, 12:00, 16:00, and 20:00, and each shooting lasted for one hour. In specific applications, two types of acquisition devices, robots and mobile phone cameras, were used. Among them, the first type of acquisition equipment is the camera equipped on the third-generation MMX5 mobile crawling platform developed by Shihe Robot MR1000 and MR1000, and the size of the collected image is 640×480. The second type of acquisition equipment is iPhone 13mini, and the size of the collected image is 3648×2736. In order to obtain the whole process image of strawberry from bud to fruit ripening, the present invention selects the period from the appearance of the first flower of strawberry to the ripening of most fruits as the data acquisition time range, a total of two months before and after, and each shooting is two days apart. Taking into account different illumination angles, strawberry images are taken at 8:00, 12:00, 16:00, and 20:00 (indoor greenhouse light-on state) in four time periods, and each shooting lasts for one hour.

[0050] In a possible implementation, the strawberry image is taken, including: taking a picture at a level, a top view, and an upward view from 50 cm away from the strawberry growth rack, and taking a picture at a level view angle once, and taking a picture at a distance of 100 cm from the growth rack, and setting the robot to move 5 cm per second in the same direction and taking 30 pictures per second each time. In a specific application, in order to make the collected strawberry images as rich as possible, take a picture at a level, a top view, and an upward view from 50 cm away from the strawberry growth rack, and take a picture at a level view angle once at a distance of 100 cm from the growth rack. Each time the picture is taken, the robot is set to move 5 cm per second in the same direction and take 30 pictures per second. In addition, the operator further uses a handheld camera to shoot at a distance of 50 cm from the ridge of the field at a top view angle, so that the captured image covers all plants in the field. Both of the above methods are output as RGB color images and stored in JPG format.

[0051] In a possible implementation, mask annotation includes dividing strawberry images into ripe fruits, immature fruits, flowers, and backgrounds. In a specific application, in order to better and more timely pollinate flowers and pick ripe fruits, strawberry flowers and fruits are mainly divided into four categories during annotation: ripe fruits: strawberry fruits that have reached edible quality and can be picked and eaten immediately; immature fruits: strawberry fruits with a red skin coloring range of no more than 70%; flowers; background. During annotation, the first three categories are mainly annotated, while the background is not annotated because it does not need to be paid attention to during actual planting. During the specific annotation, two researchers with knowledge of strawberry planting used the open source annotation script ISAT_with_segment_anything to annotate the images and stored the samples in PNG format. Representative instances of each category are shown, in which each instance of flowers, immature fruits, ripe fruits, etc. is annotated with a separate category and mask, so this dataset is also suitable for recognition and instance segmentation tasks.

[0052] In a possible implementation, after mask annotation completes the annotation of strawberry images, a strawberry data set is formed, and some strawberry images are randomly selected from the strawberry data set to form a training set, some strawberry images form a test set, and the remaining strawberry images form a verification set. In a specific application, after completing the annotation of 1938 strawberry images, the strawberry data set produced by the present invention is formed and is recorded as StrBrySet. According to a ratio of approximately 8:1:1, 1538 images are randomly selected from StrBrySet to form a training set, 200 images are used to form a test set, and the remaining 200 images are used to form a verification set. The number of each target to be identified and segmented in the training set, verification set, and test set is counted as shown in Table 1 below:

[0053] Table 1. The number of targets in the training, validation and test sets

[0054]

[0055]

[0056] In a possible implementation, a deep and narrow backbone network includes a larger number of convolution kernels and a smaller number of convolution channels. In a specific application, in the segmentation task of strawberry flowers and fruits, although the strawberry planting scene is quite complex, there are fewer categories to be identified. Increasing the network width can obtain richer features, and increasing the depth of the network can obtain more complex features. For this reason, the present invention proposes a deep and narrow network structure to adapt to the segmentation task of the present invention. Among them, deep refers to a large number of convolution kernels, and narrow refers to a small number of convolution channels. This is because by increasing the number of convolutions, complex features can be better extracted, so that the flowers and fruits of strawberries can be better distinguished from the complex background; by reducing the number of convolution channels, the model parameters can be reduced to a large extent, so that sufficiently rich features can be obtained on the basis of increasing the depth to achieve accurate classification of a small number of categories.

[0057] The deep and narrow neural network designed by the present invention is mainly reflected in the backbone network part of LSSNet, as shown in the right half of Table 2. In addition, for the convenience of comparison, Table 2 also gives the backbone network structure of the benchmark model U-Net, as shown in the left half of Table 2. Table 2 is as follows:

[0058] Table 2 Comparison of the backbone network structure between the baseline network U-Net and the network LSSNet proposed in this paper

[0059]

[0060] As can be seen from Table 2, the feature map output size of each Stage of LSSNet is the same as that of the corresponding Stage of U-Net, but the network structure is optimized and modified. Specifically, LSSNet replaces the four 3×3 convolutions of the first two Stages of U-Net with a 7×7 convolution to obtain a larger receptive field with less computational cost. In the Stage 1 to Stage 5 of the backbone network, the depth and width combinations of U-Net are (64, 128, 256, 512, 1024) and (2, 2, 2, 2, 2), respectively; while those of LSSNet are set to (64, 64, 128, 256) and (1, 2, 2, 2). For the first convolution of Stage 3 and Stage 4 of LSSNet, in order to solve the problem of large amount of information loss caused by the fixed maximum pooling of U-Net during the downsampling process, the present invention uses a convolution with learnable parameters and a step size of 2 to replace the maximum pooling operation in U-Net.

[0061] In one possible implementation, the number of convolution kernels is 7×7, and a residual structure is introduced to connect the convolutions in pairs. In specific applications, when the network is too deep, it will lead to problems such as gradient vanishing or overfitting, thereby affecting the robustness of the network model. Compared with the baseline model, the LSSNet proposed in the present invention greatly increases the depth and replaces the pooling layer with fixed parameters with a learnable convolution, which greatly increases the risk of overfitting. To alleviate this problem, in addition to the first 7×7 convolution, the present invention further introduces a residual structure to connect the convolutions in pairs.

[0062] In a possible implementation, the edge attention mechanism includes activating the spatial information weight map by Sigmoid to generate a two-dimensional space edge attention map. In specific applications, in view of the realistic factors such as clustered growth between strawberries and being close to the background, the edge of the target to be segmented is easily blurred, and the present invention proposes a kind of edge attention algorithm mechanism to guide the model to pay more attention to the border of strawberries and obtain clear edges. Specifically, the present invention finds after experiment that the average pooling operation will cause the prominent effect of the edge to be greatly reduced. For this reason, the present invention only performs maximum pooling along the channel direction, and does not perform maximum pooling in the spatial direction, and then generates a two-dimensional space edge attention map by activating the spatial information weight map by Sigmoid.

[0063] In a possible implementation, the strawberry image includes strawberry targets of different sizes, and the pyramid pooling module is used for the edge attention mechanism to cause the lack of the model's global understanding ability. In specific applications, there will be strawberry targets of different sizes in an image, which brings challenges to strawberry segmentation. The FCN module adopted by the benchmark model U-Net does not fully consider the relationship between pixels, and therefore lacks the ability to understand the image globally, so that targets of different sizes cannot be segmented well. In addition, the edge attention proposed by the present invention will make the model pay more attention to local information, thereby causing the further lack of the model's global understanding ability. In order to better segment strawberries of different sizes, the present invention introduces a pyramid pooling module at the decoding head of LSSNet, which enhances the global information perception ability by aggregating features of different scales.

[0064] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A lightweight semantic segmentation method for complex scene strawberry flower and fruit recognition, characterized in that: The following steps are involved: S1. Obtain a dataset of strawberry images, and collect images of strawberry flowers and fruits in scenes covering different lighting, time, and plant states; S2, data processing, classifying and masking the screened strawberry images; S3, designing a deep and narrow backbone network to obtain rich and complex data features of the strawberry image; S4, guiding the model to pay more attention to the boundary of the strawberry image through the edge attention mechanism to obtain clear edges; S5, introducing a pyramid pooling module at the decoding head to perceive global information by aggregating the strawberry image features of different scales; S6. Segment the strawberry image through a convolutional neural network.

2. The lightweight semantic segmentation method for complex scene strawberry flower and fruit identification according to claim 1, characterized in that: The image acquisition of strawberry flowers and fruits in scenes covering different lighting, time and plant status includes: The period from the appearance of the first flower to the ripening of the fruit of strawberry was selected as the data collection time range, which lasted for two months, with two days between each shooting; Strawberry images were captured at four time periods: 8:00, 12:00, 16:00, and 20:00, and each capture lasted for one hour.

3. The lightweight semantic segmentation method for complex scene strawberry flower and fruit identification according to claim 2, characterized in that, The strawberry image shooting includes: shooting once at three angles of level, looking down and looking up from 50 cm away from the strawberry growth rack, and shooting once at a level angle at 100 cm away from the growth rack. During each shooting, the robot is set to move 5 cm per second in the same direction and shoot 30 pictures per second.

4. The lightweight semantic segmentation method for complex scene strawberry flower and fruit identification according to claim 1, characterized in that, The mask annotation includes dividing the strawberry image into ripe fruits, unripe fruits, flowers and background.

5. The lightweight semantic segmentation method for complex scene strawberry flower and fruit identification according to claim 4, characterized in that, After the mask annotation completes the annotation of the strawberry images, a strawberry data set is formed. Part of the strawberry images are randomly selected from the strawberry data set to form a training set, part of the strawberry images are formed to form a test set, and the remaining part of the strawberry images are formed to form a verification set.

6. The lightweight semantic segmentation method for complex scene strawberry flower and fruit identification according to claim 1, characterized in that: The deep and narrow backbone network includes a larger number of convolution kernels and a smaller number of convolution channels.

7. The lightweight semantic segmentation method for complex scene strawberry flower and fruit recognition according to claim 6, characterized in that: The number of convolution kernels is 7×7, and a residual structure is introduced to connect convolutions in pairs.

8. The lightweight semantic segmentation method for complex scene strawberry flower and fruit recognition according to claim 1, characterized in that: The edge attention mechanism includes activating a spatial information weight map through Sigmoid to generate a two-dimensional spatial edge attention map.

9. The lightweight semantic segmentation method for complex scene strawberry flower and fruit identification according to claim 1, characterized in that: The strawberry image includes strawberry objects of different sizes, and the pyramid pooling module is used to eliminate the lack of global understanding ability of the model caused by the edge attention mechanism.

10. The lightweight semantic segmentation method for complex scene strawberry flower and fruit recognition according to claim 1, characterized in that: The network architecture for strawberry image segmentation is LSSNet.

Images