Soybean pod identification method based on U-Net multi-module fusion

Abstract

The invention provides a multi-module fusion soybean pod identification method based on U-Net. The method comprises the following steps: constructing a U-Net network; obtaining a soybean pod blurred image shot in a shaking state, and sending the soybean pod blurred image to a U-Net network; carrying out four times of down-sampling on the soybean pod image through a down-sampling module, and obtaining a minimum feature map; the minimum feature map is sent to an IDAM module, and an attention feature map is obtained; four times of up-sampling are carried out on the attention characteristic pattern through an up-sampling module, and a clear soybean pod image is obtained. An IDAM module is added before up-sampling is carried out on a u-net backbone network, the ability of the network to capture correlation between soybean pod images is enhanced, and the degree of attention to a pod area is improved; the EFEM module is introduced to map the edge of the soybean pod image to obtain more accurate edge information of the pod area, so that the Dice and Recall of the model are improved, and a clear image of the big-head pod is obtained.

Description

Technical Field

[0001] The present invention belongs to the field of image semantic segmentation, and in particular relates to a soybean pod recognition method based on U-Net multi-module fusion. Background Art

[0002] Soybean harvesters play an indispensable role in soybean planting and production. Their main functions include soybean harvesting, batch cutting and briquetting. However, due to the different heights of mature soybean plants, the height of the harvester's header needs to be accurately set during the harvesting process. At present, existing soybean harvesters usually use manual estimation to adjust the header height, but this method has the problems of high subjectivity and uncertainty. Setting the header too high can easily lead to increased grain loss due to missed harvesting, while setting it too low may cause the header to shovel soil and reduce the service life of the header. In addition, due to the high background similarity of the soybean pod images obtained in the soybean field, the adjacent soybean plants cause great interference to the accurate identification of the pods. At the same time, the harvester will produce a large shake during operation, resulting in blurred and missing pod feature information. Traditional semantic networks are difficult to directly obtain pod features, making it difficult to achieve accurate segmentation of pods. Summary of the invention

[0003] In view of the problems in the prior art, the present invention provides a soybean pod recognition method based on U-Net multi-module fusion, which aims to accurately and quickly identify the position of soybean plant pods, improve harvesting efficiency and harvesting quality, and reduce the breakage rate of soybeans during the harvesting process.

[0004] A soybean pod recognition method based on multi-module fusion of U-Net, comprising the following steps:

[0005] Step 1: Construct a U-Net network through downsampling module, upsampling module, IDAM module and EFEM module;

[0006] Step 2: Obtain the blurred image of soybean pods taken in a shaking state and feed it into the U-Net network;

[0007] Step 3: Downsample the soybean pod image four times through the downsampling module and obtain the minimum feature map;

[0008] Step 4: Send the minimum feature map to the IDAM module and obtain the attention feature map;

[0009] Step 5: The attention feature map is upsampled four times through the upsampling module to obtain a clear image of soybean pods, wherein the sampled data is processed by the EFEM module before each upsampling, and the output of the EFEM module is upsampled; a jump connection is set between the first three downsampling modules and the last three upsampling modules, so that the upsampling module extracts the shallow features in the downsampling module and uses them for the upsampling operation.

[0010] Further, the IDAM module includes an adjacent position attention module and a channel attention module, and the minimum feature map is sent to the channel attention module and the adjacent position attention module respectively;

[0011] The processing process of the minimum feature map in the adjacent position attention module is:

[0012] Step 4.1: The minimum feature map passes through the column attention layer X and row attention layer U of the adjacent position attention module to obtain column output and row output respectively;

[0013] Step 4.2: The row output is processed by the softmax function and the column output, and the column features are obtained by the einsum function;

[0014] Step 4.3: The output of the column features after batch normalization (BN) layer and softmax function is passed through the einsum function together with the output of the BN layer to obtain the row features;

[0015] Step 4.4: Stack the column features and row features in channels to obtain the output feature map Aa of the channel attention module;

[0016] The processing process of the minimum feature map in the channel attention module is:

[0017] Step 4.5: First, reshape the minimum feature map to obtain the feature matrix , where N represents the product of the height H and width W of the feature map; then the feature matrix of the minimum feature map With its transposed matrix Multiply them and apply the softmax layer to get the channel attention feature map Y, and ; Then, the input minimum feature map is reshaped again and multiplied with the matrix of the channel attention feature map Y. The result is reshaped again to restore the matrix size to the minimum feature map Figure 1 Finally, the result is summed element by element with the input minimum feature map, and the output feature map is obtained. ;

[0018] Step 4.6: Output feature map Multiply the output feature map Aa by the adaptive parameters and , and add the two together to get the final output attention feature map :

[0019] Among them, the parameters and is the weight that can be gradually learned, and D is the minimum feature map.

[0020] Further: The EFEM module includes an overall nested edge detection algorithm module, a complementary feature layer, a convolution layer, a connection operation module, an attention module, and a global pooling layer. The output of the EFEM module is recorded as a feature map :

[0021] in, represents direct multiplication after expansion and reading, Indicates pixel-by-pixel multiplication;

[0022] And: ,in, represents the convolution function, Represents a join operation, represents the attention module function, It is the output of the sampled data after being processed by the overall nested edge detection algorithm module; It is the output of the minimum feature map after being processed by the complementary feature layer; is the sampling data;

[0023] Feature Map ,Will As input, it is then convolved on the channel by 1×1 convolution. Compress and reduce the width W to height H through the average pooling layer to obtain the feature map ;

[0024] Feature Map ,Will As input, through the global pooling layer, the width and height are reduced to 1, and then the obtained feature map is subjected to 1×1 convolution and ReLU function activation operation to reduce the number of feature map channels and activate the processed data at the same time. Then, 1×1 convolution and Sigmoid activation are performed again to restore the feature map to its original size and obtain the predicted probability of the target area to obtain the feature map ;

[0025] Further: The combined loss function of the U-Net network for:

[0026] in, and are the balance coefficients for balancing area loss and edge loss, i is the pixel point, c and Both represent classification, represents the probability of being classified into the target area category, Represents a constant.

[0027] The beneficial effects of the present invention are as follows: an IDAM module is added before upsampling of a u-net backbone network, thereby enhancing the network's ability to capture the correlation between soybean pod images and increasing the focus on the pod area; an EFEM module is introduced to map the edges of soybean pod images to obtain more accurate edge information of the pod area, thereby improving the Dice (accuracy of distinguishing similarities in the pod edge area) and Recall (accuracy of detecting the pod area) of the model, thereby obtaining a clear image of a large-headed pod. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is the overall framework diagram of the network of the present invention; Figure 2 It is a structural block diagram of the IDAM module in the present invention; Figure 3 It is the structural block diagram of the EFEM module in the present invention. DETAILED DESCRIPTION

[0029] The present invention is described in detail below in conjunction with the accompanying drawings. The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention. The directional terms such as left, middle, right, top, and bottom in the embodiments of the present invention are only relative concepts or are based on the normal use state of the product, and should not be considered as restrictive.

[0030] A soybean pod recognition method based on multi-module fusion of U-Net, such as Figure 1 As shown, the following steps are included:

[0031] Step 1: Construct a U-Net network through a downsampling module, an upsampling module, an IDAM module (i.e., improved dual attention module) and an EFEM module (i.e., edge feature enhancement module);

[0032] Step 2: Obtain the blurred image of soybean pods taken in a shaking state and feed it into the U-Net network;

[0033] Step 3: Downsample the soybean pod image four times through the downsampling module and obtain the minimum feature map;

[0034] Step 4: Send the minimum feature map to the IDAM module and obtain the attention feature map;

[0035] Combination Figure 2 As shown, the IDAM module includes an adjacent position attention module and a channel attention module, and the minimum feature map is sent to the channel attention module and the adjacent position attention module respectively;

[0036] The processing process of the minimum feature map in the adjacent position attention module is:

[0037] Step 4.1: The minimum feature map passes through the column attention layer X and row attention layer U of the adjacent position attention module to obtain column output and row output respectively;

[0038] Step 4.2: The row output is processed by the softmax function and the column output, and the column features are obtained by the einsum function;

[0039] Step 4.3: The output of the column features after batch normalization (BN) layer and softmax function is passed through the einsum function together with the output of the BN layer to obtain the row features;

[0040] Step 4.4: Stack the column features and row features in channels to obtain the output feature map Aa of the channel attention module;

[0041] The processing process of the minimum feature map in the channel attention module is:

[0042] Step 4.5: First, reshape the minimum feature map to obtain the feature matrix , where N represents the product of the height H and width W of the feature map; then the feature matrix of the minimum feature map With its transposed matrix Multiply them and apply the softmax layer to get the channel attention feature map Y, and ; Then, the input minimum feature map is reshaped again and multiplied with the matrix of the channel attention feature map Y. The result is reshaped again to restore the matrix size to the minimum feature map Figure 1 Finally, the result is summed element by element with the input minimum feature map, and the output feature map is obtained. ;

[0043] Step 4.6: Output feature map Multiply the output feature map Aa by the adaptive parameters and , and add the two together to get the final output attention feature map :

[0044] Among them, the parameters and is the weight that can be gradually learned, and D is the minimum feature map;

[0045] Step 5: The attention feature map is upsampled four times through the upsampling module to obtain a clear image of soybean pods, wherein the sampled data is processed by the EFEM module before each upsampling, and the output of the EFEM module is upsampled; a jump connection is set between the first three downsampling modules and the last three upsampling modules, so that the upsampling module extracts the shallow features in the downsampling module and uses them for the upsampling operation; wherein, combined with Figure 3 As shown in Figure 2, the EFEM module includes an overall nested edge detection algorithm module, a complementary feature layer, a convolutional layer, a connection operation module, an attention module, and a global pooling layer. The output of the EFEM module is referred to as a feature map. :

[0046] in, represents direct multiplication after expansion and reading, Indicates pixel-by-pixel multiplication;

[0047] And: ,in, represents the convolution function, Represents a join operation, represents the attention module function, It is the output of the sampled data after being processed by the overall nested edge detection algorithm module; It is the output of the minimum feature map after being processed by the complementary feature layer; is the sampling data;

[0048] Feature Map ,Will As input, it is then convolved on the channel by 1×1 convolution. Compress and reduce the width W to height H through the average pooling layer to obtain the feature map ;

[0049] Feature Map ,Will As input, through the global pooling layer, the width and height are reduced to 1, and then the obtained feature map is subjected to 1×1 convolution and ReLU function activation operation to reduce the number of feature map channels and activate the processed data at the same time. Then, 1×1 convolution and Sigmoid activation are performed again to restore the feature map to its original size and obtain the predicted probability of the target area to obtain the feature map ;

[0050] In addition, the combined loss function of the U-Net network for:

[0051] in, and are the balance coefficients for balancing area loss and edge loss, i is the pixel point, c is the classification, Indicates classification (the classification refers to two categories, namely, target area and non-target area;) represents the probability of being classified into the target area category, Represents a constant.

[0052] The soybean pod segmentation method proposed in the present invention, which integrates the improved dual attention mechanism and the edge feature enhancement module, uses U-net as the basic network, and adds the improved dual attention module (IDAM) and the edge feature enhancement module (EFEM) to enable the soybean pods to be accurately segmented; at the same time, a combined loss function is designed to optimize and enhance the characteristics of the soybean pod image, so as to achieve accurate segmentation of the soybean pod image recognition. Among them, the IDAM module is used to suppress the network's attention to the area outside the soybean pod image, effectively reduce background interference, and improve the network's attention to the location of the soybean pod in the image, thereby reducing the amount of calculation and improving the network recognition speed and accuracy; the EFEM module is used to obtain the edge features of the soybean pod; the module can supplement the edge feature information of the soybean pod in the masked part, so as to obtain a complete soybean pod image; the EFEM module can not only enhance the ability to capture the details of the pod, but also effectively improve the integrity and clarity of the soybean pod image when processing complex scene images. In addition, in order to balance different optimization objectives and enhance the model's ability to learn specific features, multiple loss terms and loss weights are introduced by combining the loss function to increase the network's sensitivity to the target area, thereby improving the model's robustness and generalization ability. At the same time, the addition of weighted loss terms allows the network to focus more on feature extraction in the target area while suppressing background noise.

[0053] The shaking state of the harvester during operation was simulated in a known environment. An industrial camera was used to obtain images of mature soybean plants. The images were then input into the U-net backbone network. The convolutional layer and the maximum pooling layer were used to perform four downsampling operations to obtain the context information of the image and reduce the size of the feature map, thereby reducing the number of parameters and the amount of calculation of the model and increasing the robustness to small disturbances of the input image. In the downsampling process, the basic operation unit consists of a 3×3 convolution kernel, standardization, and activation function. First, a 3×3 convolution operation was performed to extract the local features of the input image, and then it was standardized. Each convolution operation was performed. The output of the layer is normalized to a distribution with a mean of 0 and a variance of 1, and finally activated by the ReLU activation function; after the input image is processed by the basic unit, the activation map is downsampled using maximum pooling; since the soybean field background image is complex and has a high similarity, the traditional network is easily disturbed by other soybean plants when identifying soybean pods. Therefore, when this network obtains the minimum feature map, that is, before upsampling, an improved attention mechanism module (IDAM) is added, and the minimum feature map is input into the module. The module suppresses irrelevant areas of the input image and highlights the target features.

[0054] To solve the problem of blurred and missing feature information of bean pods, a feature enhancement module (EFEM) based on the HED algorithm can enhance the features of the pods and supplement the missing areas. The EFEM module is added in the four upsampling processes to connect the feature map obtained from the previous layer with the feature map obtained by the HED algorithm to obtain the output feature map. At the same time, the feature map is used as input to compress and fuse in channels and space to output the image. In addition, in the image segmentation process, there is usually a problem of class imbalance, that is, some categories have fewer pixels, while other categories have more pixels. This imbalance will cause the model to be more inclined to predict the category with more pixels and ignore the features of the category with fewer pixels. To solve this problem, we used the Dice loss function to balance the importance of each category. However, when processing soybean pod images, we found that the feature of blurred image edges has a greater impact on the Dice loss function. Therefore, a combined loss function L is proposed to better focus on image edge information and improve the performance of the segmentation model.

[0055] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A soybean pod recognition method based on multi-module fusion of U-Net, characterized in that: The following steps are involved: Step 1: Construct a U-Net network through downsampling module, upsampling module, IDAM module and EFEM module; Step 2: Obtain the blurred image of soybean pods taken in a shaking state and feed it into the U-Net network; Step 3: Downsample the soybean pod image four times through the downsampling module and obtain the minimum feature map; Step 4: Send the minimum feature map to the IDAM module and obtain the attention feature map; Step 5: The attention feature map is upsampled four times through the upsampling module to obtain a clear image of soybean pods, wherein the sampled data is processed by the EFEM module before each upsampling, and the output of the EFEM module is upsampled; a jump connection is set between the first three downsampling modules and the last three upsampling modules, so that the upsampling module extracts the shallow features in the downsampling module and uses them for the upsampling operation.

2. The soybean pod recognition method based on multi-module fusion of U-Net according to claim 1 is characterized in that: The IDAM module includes an adjacent position attention module and a channel attention module, and the minimum feature map is sent to the channel attention module and the adjacent position attention module respectively; The processing process of the minimum feature map in the adjacent position attention module is: Step 4.1: The minimum feature map passes through the column attention layer X and row attention layer U of the adjacent position attention module to obtain column output and row output respectively; Step 4.2: The row output is processed by the softmax function and the column output, and the column features are obtained by the einsum function; Step 4.3: The output of the column features after batch normalization (BN) layer and softmax function is passed through the einsum function together with the output of the BN layer to obtain the row features; Step 4.4: Stack the column features and row features in channels to obtain the output feature map Aa of the channel attention module; The processing process of the minimum feature map in the channel attention module is: Step 4.5: First, reshape the minimum feature map to obtain the feature matrix , where N represents the product of the height H and width W of the feature map; then the feature matrix of the minimum feature map With its transposed matrix Multiply and apply the softmax layer to get the channel attention feature map Y. ; Then, the input minimum feature map is reshaped again and multiplied with the matrix of the channel attention feature map Y. The result is reshaped again to restore the matrix size to be consistent with the minimum feature map; finally, the result is summed element by element with the input minimum feature map; And get the output feature map ; Step 4.6: Output feature map Multiply the output feature map Aa by the adaptive parameters and , and add the two together to get the final output attention feature map : Among them, the parameters and is the weight that can be gradually learned, and D is the minimum feature map.

3. The soybean pod recognition method based on multi-module fusion of U-Net according to claim 1 is characterized in that: The EFEM module includes an overall nested edge detection algorithm module, a complementary feature layer, a convolutional layer, a connection operation module, an attention module, and a global pooling layer. The output of the EFEM module is referred to as a feature map. : in, represents direct multiplication after expansion and reading, Indicates pixel-by-pixel multiplication; for: ,in, represents the convolution function, Represents a join operation, represents the attention module function, It is the output of the sampled data after being processed by the overall nested edge detection algorithm module; It is the output of the minimum feature map after being processed by the complementary feature layer; is the sampling data; Feature Map ,Will As input, it is then convolved on the channel by 1×1 convolution. Compress and reduce the width W to height H through the average pooling layer to obtain the feature map ; Feature Map ,Will As input, through the global pooling layer, the width and height are reduced to 1, and then the obtained feature map is subjected to 1×1 convolution and ReLU function activation operation to reduce the number of feature map channels and activate the processed data at the same time. Then, 1×1 convolution and Sigmoid activation are performed again to restore the feature map to its original size and obtain the predicted probability of the target area to obtain the feature map .

4. The soybean pod recognition method based on multi-module fusion of U-Net according to claim 1 is characterized in that: Combined loss function of U-Net network for: in, and are the balance coefficients for balancing area loss and edge loss, is the pixel point, c and Both represent classification, represents the probability of being classified into the target area category, Represents a constant.

Images