A method and device for apple image segmentation based on improved U-net

Abstract

The present invention discloses an apple image segmentation method and device based on improved U‑net. The method comprises: obtaining an apple image in a real scene and preprocessing it, and using the preprocessed apple image data as a data set; training an apple image segmentation network through the data set to obtain a trained apple image segmentation network; the apple image segmentation network is an improved U‑net network; segmenting the apple image through the trained apple image segmentation network to obtain a segmentation result of the apple image. The present invention constructs a new apple image segmentation network, which uses a ResNet50 network and transfer learning to achieve multi-layer feature reuse, reducing network training time; using a residual structure to avoid gradient vanishing, and an improved semantic alignment structure to reconstruct information on the apple image, so as to better obtain detailed information of the apple, greatly improving the accuracy of apple image segmentation.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of image segmentation, and in particular to an apple image segmentation method and device based on an improved U-net. Background Art

[0002] At present, apples are mainly harvested manually, and the strong seasonal labor demand for manual harvesting easily leads to labor shortages and high labor costs. Mechanical harvesting is a widespread harvesting method that uses a shaking and capturing system to apply vibrations to tree trunks or branches to collect fruit crops. Compared with traditional manual picking, mechanical harvesting has the advantages of high efficiency and applicability to large areas. However, it often causes damage to apples and reduces the value of apples. Therefore, technological innovation is sought to reduce dependence on manual labor and reduce apple damage, and robots are used for selective picking. Machine vision systems are crucial to improving the efficiency of robot picking. Image segmentation is a key task in machine vision, and robot selective picking relies on image segmentation.

[0003] Existing technologies, such as the patent document CN2020114271911, disclose a method and system for green apple image segmentation based on a U-Net network. This method only considers the image segmentation of green apples and is not universal. At the same time, the accuracy of segmenting green apple images is not disclosed. Technical literature (Fan Yishan. Research on semantic segmentation method of apple images based on deep learning [J]. Smart Agriculture Guide, 2023, 3(16): 5-10.) proposed a method for segmenting apple images based on the Deeplabv3+ network structure. The apple images taken in real scenes were not segmented, and the error was large, which was not universal. Summary of the invention

[0004] In order to solve the problems in the prior art, the present invention provides an apple image segmentation method and device based on an improved U-net to achieve fast and accurate segmentation of apple images taken in real scenes.

[0005] In a first aspect, the present invention provides an apple image segmentation method based on an improved U-net, comprising:

[0006] S1, obtain apple images in real scenes and preprocess them, and use the preprocessed apple image data as a data set;

[0007] S2. Training an apple image segmentation network using the data set to obtain a trained apple image segmentation network; the apple image segmentation network is an improved U-net network;

[0008] S3. Segment the apple image using the trained apple image segmentation network to obtain the segmentation result of the apple image.

[0009] Optionally, the S1 specifically includes:

[0010] S11: According to the environment of the apple orchard and the maturity of the apples, apples of different maturity are photographed at different time periods to obtain original apple images;

[0011] S12: labeling the original apple image;

[0012] S13: Flip the annotated apple image horizontally and vertically to obtain the enhanced dataset;

[0013] S14: Divide the dataset into training set, validation set and test set according to the proportion.

[0014] Optionally, the apple image segmentation network is based on a U-net network, including an encoding part, a skip connection, and a decoding part connected in sequence;

[0015] Among them, the backbone network of the encoding part is the ResnNet50 network, and the trained weights are added to the ResNet50 network through transfer learning. The jump connection is an improved semantic alignment structure. The decoding part includes a residual network and an upsampling module.

[0016] Optionally, the improved semantic alignment structure introduces a CBAM attention mechanism, improves the Gaussian function of the local channel and adds a residual network to its global channel.

[0017] Optionally, the S2 specifically includes:

[0018] Perform preliminary feature extraction on the data set through the encoding part to obtain preliminary features;

[0019] Sending the preliminary features into an improved semantic alignment structure for information reconstruction to obtain a reconstructed feature map;

[0020] The preliminary features and the reconstructed feature map are transmitted to the decoding part so that the decoding part can perform network training.

[0021] Optionally, the preliminary features are sent to an improved semantic alignment structure for information reconstruction, including:

[0022] The preliminary features F∈R extracted from the encoding part C*H*W The reconstructed matrix D(F)∈R is obtained by reorganizing the global channel information structure and the local channel information structure. C*H*W , the preliminary features F∈R C*H*W The reconstructed feature map F″ is obtained through calculation of the CBAM attention mechanism;

[0023] Preliminary features F∈R C*H*WThe feature map F reconstructed by the improved semantic alignment structure, ′ The expression is:

[0024]

[0025] In the formula, represents element-by-element multiplication, H represents the height of each frame of apple picture, W represents the width of each frame of apple picture, and C represents the number of channels of each frame of apple picture.

[0026] Optionally, during the training process of S2, the distance between the output result of each part and the label result is calculated by the cross entropy loss function, and the losses of each part are superimposed by different weights;

[0027] At the same time, the parameters in the network are adjusted by using the Adam optimization algorithm and automatic learning rate through the gradient back-propagation method to obtain a trained apple image segmentation network;

[0028] Among them, the cross entropy loss function expression is:

[0029]

[0030] Among them, n is the number of categories, batch is the batch size, and y ij is the true distribution, Output distribution for the network.

[0031] In a second aspect, an embodiment of the present invention further provides an apple image segmentation device based on an improved U-net, comprising:

[0032] The data set acquisition module is used to acquire apple images in real scenes and preprocess them, and use the preprocessed apple image data as the data set;

[0033] A training module, used for training the apple image segmentation network using the data set to obtain a trained apple image segmentation network; the apple image segmentation network is an improved U-net network;

[0034] The segmentation module is used to segment the apple image through the trained apple image segmentation network to obtain the segmentation result of the apple image.

[0035] Beneficial effects of the present invention:

[0036] 1. The present invention constructs a new apple image segmentation network, which is based on the U-net network. ResNet50 is used to replace the backbone network of the U-net network, and transfer learning is introduced to realize multi-layer feature reuse, reducing the network training time. The residual network is used instead of ordinary convolution in the decoding part, thereby improving the segmentation accuracy of the network.

[0037] 2. The present invention adds an improved semantic alignment structure at the jump connection of the U-net network. The improved semantic alignment structure introduces the CBAM attention mechanism, improves the Gaussian function of the local channel, and adds a residual network to its global channel to reconstruct the information of the apple image, thereby improving the segmentation accuracy of the network.

[0038] 3. The segmentation network in the present invention captures information through two different dimensions of the apple image, namely the horizontal space and the vertical space, and adaptively adjusts the feature fusion weights of different pixels from the channel and the spatial position, so as to better obtain the detailed information of the apple and improve the accuracy of the apple image segmentation.

[0039] 4 The present invention is not only applicable to apple picking robots, but also has reference value for the visual systems of other agricultural robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A flowchart of an apple image segmentation method based on an improved U-net provided in an embodiment of the present invention;

[0041] Figure 2 The images of apples of different maturity and the corresponding annotation schematic diagrams provided by the embodiment of the present invention;

[0042] Figure 3 A schematic diagram of the network structure of an apple image segmentation network CS_U-net provided in an embodiment of the present invention;

[0043] Figure 4 A residual network structure diagram provided by an embodiment of the present invention;

[0044] Figure 5 An improved semantic alignment structure diagram provided by an embodiment of the present invention;

[0045] Figure 6 A CBAM attention mechanism structure diagram provided for an embodiment of the present invention;

[0046] Figure 7 A performance comparison chart of different segmentation methods provided by an embodiment of the present invention;

[0047] Figure 8 A comparison chart of the effects of segmenting an apple image using different methods provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0049] Example

[0050] Figure 1 A flowchart of an apple image segmentation method based on an improved U-net provided in an embodiment of the present invention specifically includes the following steps:

[0051] S1. Obtain apple images in real scenes and preprocess them, and use the preprocessed apple image data as a data set.

[0052] The above S1 specifically includes:

[0053] S11: According to the environment of the apple orchard and the maturity of the apples, apples of different maturity are photographed at different time periods to obtain original apple images.

[0054] Among them, different time periods may include morning, noon and evening; different degrees of maturity include unripe apples, ripe apples, half-ripe apples, etc.

[0055] Specifically, during the data collection process, a camera was used to shoot apple trees and move along the tree rows. This embodiment moved slowly at a speed of about 1m / s, and then extracted an image every 5s. This shooting method can enable the camera to reduce the motion blur effect. In addition, data collection can be spread over multiple days to obtain more information under different lighting conditions.

[0056] S12: Label the original apple image.

[0057] You can use annotation tools to annotate the original apple image. For example, you can use labelme to annotate the original apple image, and do not annotate the apples that cannot be distinguished by the naked eye. Figure 2 , Figure 2 Apples of different colors and stages of the ripening cycle are shown along with corresponding labeled images.

[0058] S13: The annotated apple images are flipped horizontally and vertically to obtain the enhanced dataset.

[0059] S14: Divide the dataset into training set, validation set and test set in a ratio of 8:1:1.

[0060] S2. Training an apple image segmentation network using the data set to obtain a trained apple image segmentation network; the apple image segmentation network is an improved U-net network.

[0061] See also Figure 3The apple image segmentation network CS_U-net in this embodiment is based on the U-net network, and includes an encoding part (Encoder), a skip connection, and a decoding part (Decoder) connected in sequence.

[0062] Specifically, the backbone network of the encoding part is the ResnNet50 network, and the trained weights are added to the ResNet50 network through transfer learning. The jump connection is an improved semantic alignment structure, and the decoding part includes a residual network and an upsampling module.

[0063] Furthermore, the apple image segmentation network is trained through the CS_U-net network, including the following steps:

[0064] The encoding part performs preliminary feature extraction on the data set to obtain preliminary features; the preliminary features are sent to the improved semantic alignment structure for information reconstruction to obtain a reconstructed feature map; the preliminary features and the reconstructed feature map are transmitted to the decoding part for network training by the decoding part.

[0065] Among them, the backbone network ResNet50 of the encoding part has stronger feature extraction capabilities. The ResNet50 network designs the residual network structure to increase the network depth. For details, see Figure 4 ,The residual network structure diagram avoids the problem that the gradient is more likely to disappear as the network is farther away from the loss layer by propagating across layers.

[0066] See also Figure 5 , Figure 5 Figure 1 is a schematic diagram of the improved semantic alignment structure in the CS_U-net network, which is used to enhance the extraction of small objects. The improved semantic alignment structure combines the global channel information distribution reconstruction matrix and the local channel information distribution reconstruction to ensure the integrity of the global semantic information.

[0067] The information of apple images can be reconstructed through this structure, which can improve the segmentation effect of apple images. In order to enhance the segmentation ability of small objects, the CBAM attention mechanism is introduced in the improved semantic alignment structure, the amplitude of the local channel Gaussian function and the flexibility of mapping are improved, and the residual network is added to its global channel.

[0068] See also Figure 6 CBAM consists of a channel attention module and a spatial attention module, where the channel attention can adjust the feature representation of different channels, and the spatial attention can adjust the key information of different spatial positions. Input feature map F∈R C*H*W After one-dimensional convolution, output M C ∈R C*1*1 , MC Multiplying by F gives Input F′ into the spatial attention module for two-dimensional convolution to obtain M s ∈R 1*H*W , M s Multiplying by F′ gives Among them, F is the preliminary feature extracted from the encoding part.

[0069] Based on the improved semantic alignment structure, the preliminary features extracted by the encoding module are reconstructed, including: the preliminary features F∈R extracted by the encoding part C*H*W The reconstructed matrix D(F)∈R is obtained by reorganizing the global channel information structure and the local channel information structure. C*H*W , the preliminary features F∈R C*H*W The reconstructed feature map F′′ is obtained through calculation of the CBAM attention mechanism;

[0070] Preliminary features F∈R C*H*W Feature map F reconstructed by improved semantic alignment structure ， The expression is:

[0071]

[0072] In the formula, represents element-by-element multiplication, H represents the height of each frame of apple pictures, W represents the width of each frame of apple pictures, and C represents the number of channels of each frame of apple pictures. The CBAM attention mechanism enhances the ability to extract features from semantically aligned structures, and the residual mechanism performs gradient optimization.

[0073] Channel information distribution matrix D(F)∈R C*H*W Reconstruct the matrix D from the global channel information distribution h (F)∈R C*1*1 And the local channel information distribution reconstruction matrix D C (F)∈R C*1*1 The improved semantic alignment structure can extract global semantic information and local semantic information. The calculation formula of the matrix D(F) is:

[0074]

[0075] Where: represents element-by-element multiplication, and β is the Tanh activation function. After the activation function, the dimension of D(F) is changed from R C *1*1 Extension to R C*H*W .

[0076] The input feature map F is subjected to global average pooling to compress the semantic information, which is conducive to the convolutional neural network to learn and adjust the semantic features of the feature map. The compressed semantic features are F ab ∈R C*1*1 , the expression is:

[0077]

[0078] Where: F(i,J) represents the value of the feature map pixel (i,J).

[0079] A mechanism similar to SENet is used to reconstruct the global channel information distribution, introduce weight ω for each layer of semantic features, and introduce a residual network to smoothly reconstruct channel features, which is beneficial to network gradient optimization. The mapping relationship between the global channel reconstruction matrix and semantic information is established through gradient update, and the formula is:

[0080]

[0081] In the formula, CW is the residual network, is a 1×1 convolution (the input channel is 2048, the output channel is 1024, and the bias term is False), BN is a batch normalization layer, ReLU is the activation function, and δ(·) is a 1×1 convolution (the input channel is 1024, the output channel is 2048, and the bias term is False).

[0082] The distribution of local semantic information conforms to the Gaussian distribution. The Gaussian function is used as the mapping function to construct the relationship between the input information and the local information reconstruction matrix. The Gaussian distribution function equation is:

[0083]

[0084] Where: μ is the population mean; σ 2 is the overall variance. In order to enhance the robustness of the mapping relationship, the input feature information is normalized. A new variable is defined to adapt to the changes in the feature map:

[0085]

[0086] Substituting equation (6) into equation (5), we can obtain the simplified Gaussian distribution function:

[0087]

[0088] In order to make the mapping relationship more flexible, the variable γ∈[1,3] is introduced to increase the flexibility of the mapping, and the amplitude of the Gaussian distribution function is expanded. The transformed Gaussian distribution function can be expressed as:

[0089]

[0090] From equations (3), (6), (7), and (8), we can get the mapping relationship between the local reconstruction matrix and the semantic information:

[0091]

[0092] Furthermore, in the process of training the CS_U-net network, the cross entropy loss function is used to calculate the distance between the output result of each part and the label result, and the losses of each part are superimposed through different weights for subsequent training; at the same time, the parameters in the network are adjusted by the Adam optimization algorithm and the automatic learning rate through the gradient back propagation method to obtain the trained CS_U-net network;

[0093] The cross entropy loss function expression used is as follows:

[0094]

[0095] Where n is the number of categories, batch is the batch size, and y ij is the true distribution, Output distribution for the network.

[0096] S3. Segment the apple image using the trained apple image segmentation network to obtain the segmentation result of the apple image.

[0097] In this embodiment, the best network is selected as the CS_U-net network for segmentation based on the Mean Intersection over Union (MiOU), precision, recall, and F1-score of the training.

[0098] Furthermore, the mean intersection over union (MiOU), precision, recall, and F1-score of the CS_U-net network model in this example are compared with those of the existing models U-net, SegNet, PsPnet, HRNet, and Deeplabv3+. Figure 7 The results show that among the U-net, SegNet, PsPnet, HRNet, and Deeplabv3+ models, the U-net has the best performance. The Miou, Precision, Recall, and F1-score of the CS_U-net network model are 3.58%, 9%, 8%, and 8.48% higher than those of the U-net model, respectively.

[0099] In this example, the CS_U-net network model is compared with the existing model in the segmentation of apple images. Figure 8 As shown, Figure 8 (a) is listed as an apple image, Figure 8 (bg) Columns are the segmentation results of CS_U-net, Deeplabv3+, HRNet, PsPnet, SegNet, and U-net. The red rectangular box area in the figure is the segmentation effect comparison area of ​​each model. In the apple concentration area in the first row, the leaf occlusion area in the second row, and the small apple area in the third row, the CS_U-net model has the best segmentation effect. Therefore, it shows that the method proposed in the present invention solves the problem of small apple target information loss and improves the segmentation effect of the network.

[0100] The embodiment of the present invention further provides an apple image segmentation device based on an improved U-net, comprising:

[0101] The data set acquisition module is used to acquire apple images in real scenes and preprocess them, and use the preprocessed apple image data as the data set;

[0102] A training module, used for training the apple image segmentation network using the data set to obtain a trained apple image segmentation network; the apple image segmentation network is an improved U-net network;

[0103] The segmentation module is used to segment the apple image through the trained apple image segmentation network to obtain the segmentation result of the apple image.

[0104] The data set acquisition module is specifically used to execute:

[0105] S11: According to the environment of the apple orchard and the maturity of the apples, apples of different maturity are photographed at different time periods to obtain original apple images;

[0106] S12: labeling the original apple image;

[0107] S13: Flip the annotated apple image horizontally and vertically to obtain the enhanced dataset;

[0108] S14: Divide the dataset into training set, validation set and test set according to the proportion.

[0109] The apple image segmentation network CS_U-net is based on the U-net network, including an encoding part, a skip connection, and a decoding part connected in sequence;

[0110] Among them, the backbone network of the encoding part is the ResnNet50 network, and the trained weights are added to the ResNet50 network through transfer learning. The jump connection is an improved semantic alignment structure. The decoding part includes a residual network and an upsampling module.

[0111] The improved semantic alignment structure introduces the CBAM attention mechanism, improves the Gaussian function of the local channel and adds a residual network to its global channel.

[0112] The above training modules are specifically used for:

[0113] Perform preliminary feature extraction on the data set through the encoding part to obtain preliminary features;

[0114] Sending the preliminary features into an improved semantic alignment structure for information reconstruction to obtain a reconstructed feature map;

[0115] The preliminary features and the reconstructed feature map are transmitted to the decoding part so that the decoding part can perform network training.

[0116] The preliminary features are sent into the improved semantic alignment structure for information reconstruction, including:

[0117] The preliminary features F∈R extracted from the encoding part C*H*W The reconstructed matrix D(F)∈R is obtained by reorganizing the global channel information structure and the local channel information structure. C*H*W , the preliminary features F∈R C*H*W The reconstructed feature map F″ is obtained through calculation of the CBAM attention mechanism;

[0118] Preliminary features F∈R C*H*W Feature map F reconstructed by improved semantic alignment structure ， The expression is:

[0119]

[0120] In the formula, represents element-by-element multiplication, H represents the height of each frame of apple picture, W represents the width of each frame of apple picture, and C represents the number of channels of each frame of apple picture.

[0121] Optionally, in the training module, the distance between the output result of each part and the label result is calculated by the cross entropy loss function, and the losses of each part are superimposed by different weights;

[0122] At the same time, the parameters in the network are adjusted by using the Adam optimization algorithm and automatic learning rate through the gradient back-propagation method to obtain a trained apple image segmentation network;

[0123] Among them, the cross entropy loss function expression is:

[0124]

[0125] Among them, n is the number of categories, batch is the batch size, and y ij is the true distribution, Output distribution for the network.

[0126] An improved U-net apple image segmentation device provided in an embodiment of the present invention can execute an improved U-net apple image segmentation method provided in any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0127] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for apple image segmentation based on improved U-net, characterized in that: include: S1, obtain apple images in real scenes and preprocess them, and use the preprocessed apple image data as a data set; S2. Training an apple image segmentation network using the data set to obtain a trained apple image segmentation network; the apple image segmentation network is an improved U-net network; S3, segmenting the apple image through the trained apple image segmentation network to obtain the segmentation result of the apple image; The apple image segmentation network is based on a U-net network, including an encoding part, a skip connection, and a decoding part connected in sequence; The backbone network of the encoding part is the ResnNet50 network, and the trained weights are added to the ResNet50 network through transfer learning. The jump connection is an improved semantic alignment structure. The decoding part includes a residual network and an upsampling module. The improved semantic alignment structure introduces the CBAM attention mechanism, improves the Gaussian function of the local channel and adds a residual network to its global channel.

2. The method according to claim 1, characterized in that The S1 specifically includes: S11: According to the environment of the apple orchard and the maturity of the apples, apples of different maturity are photographed at different time periods to obtain original apple images; S12: labeling the original apple image; S13: Flip the labeled apple images horizontally and vertically to obtain the enhanced dataset; S14: Divide the dataset into training set, validation set and test set according to the proportion.

3. The method according to claim 1, characterized in that The S2 specifically includes: Perform preliminary feature extraction on the data set through the encoding part to obtain preliminary features; Sending the preliminary features into an improved semantic alignment structure for information reconstruction to obtain a reconstructed feature map; The preliminary features and the reconstructed feature map are transmitted to the decoding part so that the decoding part can perform network training.

4. The method according to claim 3, characterized in that The preliminary features are sent to the improved semantic alignment structure for information reconstruction, including: The preliminary features F∈R extracted from the encoding part C*H*W The reconstructed matrix D(F)∈R is obtained by reorganizing the global channel information structure and the local channel information structure. C*H*W , the preliminary features F∈R C*H*W The reconstructed feature map F″ is obtained through calculation of the CBAM attention mechanism; Preliminary features F∈R C*H*W Feature map F reconstructed by improved semantic alignment structure ′ The expression is: In the formula, represents element-by-element multiplication, H represents the height of each frame of apple picture, W represents the width of each frame of apple picture, and C represents the number of channels of each frame of apple picture.

5. The method according to claim 1, characterized in that During the training process of S2, the distance between the output result of each part and the label result is calculated through the cross entropy loss function, and the losses of each part are superimposed through different weights; At the same time, the parameters in the network are adjusted by using the Adam optimization algorithm and automatic learning rate through the gradient back-propagation method to obtain a trained apple image segmentation network; Among them, the cross entropy loss function expression is: Among them, n is the number of categories, batch is the batch size, and y ji is the true distribution, Output distribution for the network.

6. An apple image segmentation device based on an improved U-net, characterized in that: include: The data set acquisition module is used to acquire apple images in real scenes and preprocess them, and use the preprocessed apple image data as the data set; A training module, used for training the apple image segmentation network using the data set to obtain a trained apple image segmentation network; the apple image segmentation network is an improved U-net network; A segmentation module is used to segment the apple image using a trained apple image segmentation network to obtain a segmentation result of the apple image; The apple image segmentation network is based on a U-net network, including an encoding part, a skip connection, and a decoding part connected in sequence; The backbone network of the encoding part is the ResnNet50 network, and the trained weights are added to the ResNet50 network through transfer learning. The jump connection is an improved semantic alignment structure. The decoding part includes a residual network and an upsampling module. The improved semantic alignment structure introduces the CBAM attention mechanism, improves the Gaussian function of the local channel and adds a residual network to its global channel.

Images