A corn identification method for soybean-corn intercropping mode

Abstract

A corn recognition method for a soybean-corn interplanting mode comprises the following steps: obtaining an original image; subjecting the original image to a convolution module to obtain a feature map F1; processing the original image through a boundary texture coordination module to obtain a feature map Fb; subjecting the original image to a convolution module, five downsamplings and five relative position encoding modules to obtain a feature map, and sequentially subjecting the feature map to a convolution module and a convolution transposition module, and then performing a series operation on the output of the feature map Fb and the result of the original image after being processed by two convolutions, so as to obtain a prediction result; and enhancing the texture boundary of the target feature by introducing three modules, namely, a relative position encoding module, a boundary texture coordination module and a multi-scale feature fusion module, and fully considering the use of shallow features in a network structure, thereby improving the accuracy of the details of the recognition result, so that the present invention performs better in image recognition tasks.

Description

Technical Field

[0001] The invention belongs to the field of image segmentation, and in particular relates to a corn recognition method used in a soybean-corn interplanting mode. Background Art

[0002] When the combine harvester is operating, the recognition accuracy of corn ears affects the operating efficiency and harvesting loss rate of the harvester. By providing real-time feedback on the recognition results of corn ears, the synchronous operation of the header device and the harvester body is realized, so that the combine harvester can more accurately adjust the operating parameters such as harvesting height and speed to adapt to different crops and growth conditions, thereby improving the operating efficiency. In the actual harvesting environment, as the light intensity changes, the color, shape and other characteristics of the corn ears may change, resulting in a decrease in recognition accuracy. In addition, the growth characteristics of corn ears vary due to differences in climate, soil and other factors, resulting in a decrease in detection accuracy. At the same time, there may be mutual occlusion between corn plants, making some corn ears difficult to identify. Finally, since the operation of the harvester will cause straw impurities to splash, the characteristic information of the corn ears is interfered with during the recognition process, thereby affecting the recognition accuracy and efficiency.

[0003] Traditional methods have also made some contributions to solve the above problems. In order to accurately identify corn and interference factors in complex field scenes, an improved dual-attention semantic segmentation method is proposed to achieve corn recognition and fine segmentation by obtaining morphological boundaries. In order to accurately measure the geometric features of corn, a three-region image enhancement model is constructed, and the enhanced image is segmented using morphological image processing methods. In order to obtain the position information of corn ears in the image, a bias vector is introduced in the relative position encoding to automatically learn the relative position information of time nodes when linearly mapping the feature tensor. In order to enhance the semantic information of shallow features of the network, an end-to-end attention and feature fusion SSD network is proposed. In order to maintain the high resolution and style of the original image, a multi-scale patch generation adversarial network with edge detection image restoration is proposed.

[0004] The above methods all have excellent performance in dealing with interference factors in the corn ear recognition process, but they are limited to static corn ear recognition and cannot meet the dynamic recognition requirements of combine harvesters. In addition, in the harvesting process, faced with interference from factors such as straw impurities splashing and harvester shaking, existing algorithms are difficult to accurately identify corn ears in such an environment. Summary of the invention

[0005] In response to the problems in the prior art, the present invention provides a corn recognition method for soybean-corn intercropping mode, which aims to restore lost information and improve recognition accuracy, ensure that while extracting deep features of the image, full consideration is given to the utilization of shallow features in the network structure, and improve the accuracy of the details of the segmentation results.

[0006] A corn identification method for soybean-corn interplanting mode comprises the following steps:

[0007] Step 1: Get the original image;

[0008] Step 2: After the original image passes through the convolution module, the feature map F1 is obtained;

[0009] Step 3: Process the original image through the boundary texture coordination module to obtain the feature map Fb;

[0010] The boundary texture coordination module includes a maximum pooling layer, a skip connection module and a bidirectional feature fusion module. The original image is sent to the bidirectional feature fusion module after the maximum pooling layer. The bidirectional feature fusion module converts the feature map The output after feature fusion with the output of the maximum pooling layer is then concatenated with the result of the original image after being processed by the skip connection module to obtain the feature map Fb;

[0011] Step 4: After the original image passes through the convolution module, 5 downsampling and 5 relative position encoding modules, the feature map is obtained. , and the feature map The outputs after the convolution module and the convolution transpose module are sequentially connected in series with the feature map Fb and the result of the original image after two convolutions to obtain the prediction result.

[0012] Further:

[0013] In step 4, according to the original feature map Contains the location information of each pixel With feature information , after downsampling and relative position encoding of the backbone network, the feature vector of the central pixel of any local space in the feature map is obtained , obtain the area where the corresponding pixel point set representing the target feature corn ear in the image is located;

[0014] First, each pixel is concatenated through the local spatial coding unit Location information With feature information , using the proximity search algorithm to obtain pixel points Around Nearby points With feature information ;

[0015] Secondly, use a multi-layer perceptron to calculate the pixel position Encoding is performed, and the encoding points near the pixel points are represented as follows:

[0016]

[0017] in, Represents the distance and direction relationship between the adjacent points and the center point, It is a serial operation. That is, the adjacent position code point, Used to obtain the position encoding tensor;

[0018] Then, concatenate the adjacent position code points and the adjacent position code point features ,Right now As the output result of local space encoding;

[0019] Finally, construct a scoring function , calculate the score value of each adjacent pixel around the target feature ;

[0020]

[0021]

[0022] in, is the weight of each neighboring point feature, is the feature of each adjacent point. According to the score function, the score value of each adjacent point is calculated, and the feature vector of the central pixel is obtained by weighted summation. .

[0023] Further: In step 3, first, the feature map The pixel representation of the target feature boundary area is redefined, and the area defined in the boundary texture feature map is Inner center pixel Four weeks The feature vector of the size range is represented as ;

[0024]

[0025]

[0026] in, Indicates capacity The matrix Any pixel within and the center pixel difference; Represents pixels around Neighborhood of size;

[0027] Secondly, the feature vector After conversion, the difference between the center pixel of the target feature boundary area and other pixels without occlusion is expressed as ;

[0028]

[0029] in, is a hyperparameter, Representation characteristics Appearance data;

[0030] Then, a binary boundary mask is introduced for boundary feature prediction After that, Defined as the pixel is regarded as an evaluation index of boundary features, and the prediction score is expressed as :

[0031]

[0032] Finally, the prediction result of the boundary feature is made opposite to the expected direction, so that the image can fully reflect the characteristics of the corn ear;

[0033] Further: Step 4 is specifically to perform multi-scale fusion of deep features and shallow features in the network through a multi-scale feature fusion module. The multi-scale feature fusion module consists of three branches. Branch 1 represents the deep feature map of the original image after being processed by the backbone network, that is, the feature map ; Branch 2 represents the original image and feature map The feature map after processing by the boundary feature texture coordination module is the feature map Fb; branch three is the shallow feature map obtained after the original image passes through two convolution blocks; the feature maps of the three branches are added element by element to realize the fusion of deep features and shallow features in the network to obtain preliminary prediction results.

[0034] Further: Also includes step 5: setting the combination function :

[0035]

[0036] in,

[0037]

[0038]

[0039]

[0040]

[0041] in, is the weight used to handle the prediction problem, is the regulating factor, represents the prediction score, is an adjustable focus parameter and is greater than zero, is the prediction score of the interference feature by the multi-scale feature fusion module, It is a feature The true category label of is the prediction score of the target category feature by the multi-scale feature fusion module; when the feature map passes through the multi-scale feature fusion module, features of multiple scales are extracted and these features are defined as , where each eigenvector It is The characteristics of the scale, is the weight vector obtained through training for interfering with the linear transformation of features, It is the weight vector obtained through training for the linear transformation of target category features.

[0042] The beneficial effects of the present invention are as follows: by introducing three modules, namely, a relative position encoding module, a boundary texture coordination module and a multi-scale feature fusion module, the texture boundary of the target feature is enhanced, and full consideration is given to the use of shallow features in the network structure, thereby improving the accuracy of the details of the recognition result, making the present invention perform better in image recognition tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a schematic diagram of the overall network model of the present invention;

[0044] Figure 2 It is a schematic diagram of the relative position encoding module in the present invention;

[0045] Figure 3 Schematic diagram of a multi-scale feature fusion module in the present invention;

[0046] Figure 4 It is a schematic diagram of data processing in the boundary texture coordination module of the present invention. DETAILED DESCRIPTION

[0047] The present invention is described in detail below in conjunction with the accompanying drawings. The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention. The directional terms such as left, middle, right, top, and bottom in the embodiments of the present invention are only relative concepts or are based on the normal use state of the product, and should not be considered as restrictive.

[0048] A corn identification method for soybean-corn intercropping mode, such as Figure 1 As shown, the following steps are included:

[0049] Step 1: Get the original image;

[0050] Step 2: After the original image passes through the convolution module (Conv-block), the feature map F1 is obtained;

[0051] Step 3: Process the original image through the Boundary Texture Coordination Module (BTCM) to obtain the feature map Fb;

[0052] Among them, combined Figure 3 and Figure 4 As shown in Figure 2, the boundary texture coordination module (BTCM) includes a maximum pooling layer (Maxpoling), a skip connection module (Skip Connection) and a bidirectional feature fusion module (concat). The original image is sent to the bidirectional feature fusion module after the maximum pooling layer. The bidirectional feature fusion module converts the feature map The output after feature fusion with the output of the maximum pooling layer is then concatenated with the result of the original image after being processed by the skip connection module to obtain the feature map Fb;

[0053] Specifically, in step 3, the boundary feature texture coordination module is responsible for processing the boundary texture features of the corn ear and coordinating the prediction results of interference features such as occluded straw impurities, so that the image can reflect the complete features of the corn ear;

[0054] First, the feature map The pixel representation of the target feature boundary area is redefined, and the area defined in the boundary texture feature map is Inner center pixel Four weeks The feature vector of the size range is represented as ;

[0055]

[0056]

[0057] in, Indicates capacity The matrix Any pixel within and the center pixel difference; Represents pixels around Neighborhood of size;

[0058] Secondly, in order to reflect the actual boundary area The appearance data of the central pixel of the target feature in the present invention converts the feature vector After conversion, the difference between the center pixel of the target feature boundary area and other pixels without occlusion is expressed as ;

[0059]

[0060] in, is a hyperparameter, Representation characteristics Appearance data;

[0061] Then, it is worth noting that the minimum difference between the center pixel of the boundary region and any pixel in the region is but in Therefore, when introducing a binary boundary mask for boundary feature prediction After that, the present invention will Defined as the pixel is regarded as an evaluation index of boundary features, and the prediction score is expressed as :

[0062]

[0063] Finally, for the predicted boundary features, the present invention finds that the boundary feature prediction score of the central pixel of the target feature is stable within a certain area; since the appearance of the boundary feature and the interference feature are similar, some prediction scores will be abnormal. At this time, the present invention expects that the prediction score of this boundary feature is close to the prediction score of the central feature boundary feature of the target feature, so that the prediction result of the boundary feature is opposite to the expected direction, and the image fully reflects the features of the corn ear;

[0064] Step 4: After the original image passes through the convolution module (Conv-block), 5 downsampling and 5 relative position encoding modules (LM), the feature map is obtained. , and the feature map The output after the convolution module (Conv-block) and the convolution transposition module (Conv2dTrans) are sequentially connected with the feature map Fb and the result of the original image after two convolutions to obtain the prediction result;

[0065] Combination Figure 2 As shown, the original image passes through the backbone extraction network to generate a feature map There are 6 feature maps containing the position information and feature information of each pixel point; the original image is passed through a convolution block to obtain the initial feature map ; Feature map Contains relatively complete feature information in the original image, that is, including the position information of each pixel in the feature map and feature information ; According to the original feature map Contains the location information of each pixel With feature information , after downsampling and relative position encoding of the backbone network, the feature vector of the central pixel of any local space in the feature map is obtained , obtain the area where the corresponding pixel point set representing the target feature corn ear in the image is located;

[0066] First, each pixel is concatenated through the local spatial coding unit Location information With feature information ,like Figure 2 As shown, to observe the local geometric pattern, a proximity search algorithm is used to obtain pixel points Around Nearby points With feature information ;

[0067] Secondly, in order to obtain local spatial features, the pixel position is analyzed using a multi-layer perceptron Encoding is performed, that is, the AC module in the figure, and the encoding points near the pixel points are expressed as follows:

[0068]

[0069] in, Represents the distance and direction relationship between the adjacent points and the center point, It is a serial operation. That is, the adjacent position code point, Used to obtain the position encoding tensor;

[0070] Then, concatenate the adjacent position code points and the adjacent position code point features ,Right now As the output result of local space encoding;

[0071] Finally, in order to enhance the effectiveness of adjacent code points in feature aggregation, the present invention constructs a scoring function , calculate the score value of each adjacent pixel around the target feature ;

[0072]

[0073]

[0074] in, is the weight of each neighboring point feature, is the feature of each adjacent point. According to the score function, the score value of each adjacent point is calculated, and the feature vector of the central pixel is obtained by weighted summation. ;

[0075] The multi-scale feature fusion module is used to fuse the deep features and shallow features in the network to achieve accurate recognition of corn ears. Figure 3 As shown in the figure, the multi-scale feature fusion module consists of three branches. Branch 1 represents the deep feature map of the original image after being processed by the backbone network, that is, the feature map , feature map Due to multiple convolution operations and relative position encoding modules, the feature map It contains the relative position relationship information of the target feature and the interference feature, but the size also changes. After a convolution block and transposed convolution, it is restored to the size of the initial feature map; Branch 2 represents the original image and feature map The feature map processed by the boundary feature texture coordination module, namely the feature map Fb, removes the interference features and contains the contour information of the target features; branch three is the shallow feature map obtained after the original image passes through two convolution blocks, which contains relatively complete feature information in the original image; the feature maps of the three branches are added element by element to realize the fusion of deep features and shallow features in the network, and obtain preliminary prediction results.

[0076] In addition, step 5 is also included: the present invention proposes a combined loss function , which is used to measure the difference between the prediction result of the model and the true value, so that the prediction result is as close to the true value as possible; the present invention combines the Hinge loss function and the Focal loss function to solve the interference problem of connective tissue. The loss function effectively reduces the weight of the target, so that the model can focus more on the interference features, thereby improving the overall classification performance of the model; setting the combination function :

[0077]

[0078] in,

[0079]

[0080]

[0081]

[0082]

[0083] in, is the weight used to handle the prediction problem, is the regulating factor, represents the prediction score, is an adjustable focus parameter and is greater than zero, is the prediction score of the interference feature by the multi-scale feature fusion module, It is a feature The true category label of is the prediction score of the target category feature by the multi-scale feature fusion module; when the feature map passes through the multi-scale feature fusion module, features of multiple scales are extracted and these features are defined as , where each eigenvector It is The characteristics of the scale, is the weight vector obtained through training for interfering with the linear transformation of features, is the weight vector obtained through training for the linear transformation of target category features; combined loss function In and The processing amplifies the difference in prediction results between the interference feature and the target feature. The greater the difference, the greater the loss function value. At the same time, the feature prediction score mentioned above is introduced. Used to adjust the weight distribution of interference features by adjusting The size of can be used to achieve weighted processing of interference features. A large number of experimental results show that as As the value increases, the training loss of the target feature decreases, and the model's attention is focused on the classification of interference features.

[0084] In the present invention, when corn ears are identified during the harvesting process, since corn plants have the same growth characteristics, the distribution of corn plant stalks, branches and leaves and corn ears has a certain relative position relationship; however, the recognition results of existing algorithms are affected by the interference features in the target area, resulting in errors between the segmentation results and the actual situation; the present invention adopts a relative position encoding module and encodes the positions of pixels through a multi-layer perceptron to capture the relative position relationship between the corn plant stalks, branches and leaves and corn ears, thereby enhancing the ability to extract target features in the image; at the same time, since the splashing of straw impurities may block the corn ears, the previous and next information of the local features of the target area in the image is incoherent, which brings great challenges to the recognition task. Therefore, the present invention proposes a boundary texture coordination module to promote the feature prediction of interfering impurities to develop in the opposite direction of the expected direction; after the above operations, the feature image has undergone multiple downsampling operations, which can reduce the amount of calculation and increase the receptive field, but will lead to reduced resolution and information loss. The information is lost, making the recognition result not fine enough; in order to restore the lost information and improve the recognition accuracy, the present invention introduces a multi-scale feature fusion module to ensure that the shallow features in the network structure are fully considered while extracting the deep features of the image, thereby improving the accuracy of the details of the segmentation result; wherein, the present invention uses a relative position encoding module, a boundary texture coordination module and a multi-scale feature fusion module to improve the FCN32s structure, wherein the relative position encoding module (LM) encodes the position of the pixel points through a multi-layer perceptron, captures the relative position relationship between the stalks, branches and leaves of the corn plant and the corn ears, and determines the area where the corn ears are located in the image; the boundary texture coordination module (BTCM) is responsible for processing the boundary texture features of the corn ears, and coordinates the prediction results of interference features such as occluded straw impurities, so that the image can reflect the complete features of the corn ears; finally, the deep features and shallow features in the network are multi-scale fused through the multi-scale feature fusion module (FM) to achieve accurate recognition of corn ears.

[0085] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A corn identification method for soybean-corn interplanting mode, characterized in that: The following steps are involved: Step 1: Get the original image; Step 2: After the original image passes through the convolution module, the feature map F1 is obtained; Step 3: Process the original image through the boundary texture coordination module to obtain the feature map Fb; Among them, the boundary texture coordination module includes a maximum pooling layer, a skip connection module and a bidirectional feature fusion module. The original image is sent to the bidirectional feature fusion module after the maximum pooling layer. The bidirectional feature fusion module performs feature fusion on the feature map F1 and the output of the maximum pooling layer, and then performs a series operation with the result of the original image processed by the skip connection module to obtain the feature map Fb; First, the pixel representation of the target feature boundary area in the feature map F1 is redefined, and the central pixel x in the region s in the boundary texture feature map is defined as i The feature vectors in the surrounding k×k range are expressed as in, Represents a matrix with capacity k×k Any pixel within and the center pixel x i Difference; K i Represents pixel x i The surrounding k×k neighborhood; Secondly, the feature vector After conversion, the difference between the center pixel of the target feature boundary area and other pixels without occlusion is expressed as Among them, δ is a hyperparameter, x i (m) Represents the appearance data of feature m; Then, a binary boundary mask b is introduced for boundary feature prediction i After that, Defined as the pixel x i It is regarded as an evaluation index of boundary features, and the prediction score is expressed as L bfp : Finally, the prediction result of the boundary feature is made opposite to the expected direction, so that the image can fully reflect the characteristics of the corn ear; Step 4: After the original image is processed by the convolution module, 5 downsampling and 5 relative position encoding modules, the feature map F6 is obtained. The output of the feature map F6 after the convolution module and the convolution transposition module is connected in series with the feature map Fb and the result of the original image after two convolutions to obtain the prediction result. Among them, according to the position information p of each pixel contained in the original feature map F1 i With feature information f i , after downsampling and relative position encoding of the backbone network, the feature vector of the central pixel of any local space in the feature map is obtained Obtaining the area where the corresponding pixel point set representing the target feature corn ear in the image is located; First, the position information p of each pixel i is concatenated through the local spatial coding unit i With feature information f i , use the proximity search algorithm to obtain the K neighboring points around pixel i With feature information {f i 1 …f i k …f i K }; Secondly, the pixel position is encoded using a multi-layer perceptron MLP, and the encoding points of the pixel proximity positions are represented as follows: in, Represents the distance and direction relationship between the adjacent points and the center point, It is a serial operation. That is, the neighboring position encoding point, MLP is used to obtain the position encoding tensor; Then, concatenate the adjacent position coding points and the adjacent position coding point features {r i 1 f i 1 …r i k f i k …r i K f i K },Right now As the output result of local space encoding; Finally, a scoring function g() is constructed to calculate the score value of each adjacent pixel around the target feature. Where W = {ω1,ω2,…,ω i } is the weight vector of each neighboring point feature, is the feature of the Kth neighboring point. According to the score function, the score value of each neighboring point is calculated, and the feature vector of the central pixel is obtained by weighted summation. The deep features and shallow features in the network are multi-scale fused through the multi-scale feature fusion module. The multi-scale feature fusion module consists of three branches. Branch one represents the deep feature map of the original image after being processed by the backbone network, that is, the feature map F6; branch two represents the feature map of the original image and the feature map F1 after being processed by the boundary feature texture coordination module, that is, the feature map Fb; branch three is the shallow feature map obtained after the original image passes through two convolution blocks; the feature maps of the three branches are added element by element to realize the fusion of the deep features and shallow features in the network, and obtain preliminary prediction results.

2. The corn identification method for soybean-corn interplanting mode according to claim 1, characterized in that: Also includes step 5: setting the combination function L FH : L FH =αL Focal +βL Hinge in, L Focal =-s t (1-L bfp ) γ log(L bfp ) Among them, σ is the weight used to deal with the prediction problem, (1-L bfp ) γ is the adjustment factor, L bfp represents the prediction score, γ is an adjustable focusing parameter and is greater than zero, s j is the prediction score of the multi-scale feature fusion module for the interference feature, y i is the true category label of feature i, and It is the prediction score of the target category feature by the multi-scale feature fusion module; when the feature map passes through the multi-scale feature fusion module, features of multiple scales are extracted, and these features are defined as f1, f2, ..., f k , where each eigenvector f i is the feature of the i-th scale, is the weight vector obtained through training for interfering with the linear transformation of features, It is the weight vector obtained through training for the linear transformation of target category features.

Images