Stalk crop height calculation method and composite harvesting method for stalk crops

Abstract

The invention provides a stalk crop height calculation method and a stalk crop composite harvesting method, and the stalk crop height calculation method comprises the following steps: shooting a long-distance A image and a short-distance B image along the same horizontal direction; performing the same processing on the image A and the image B through an NF-SAM model to obtain to-be-processed features; carrying out the deconvolution and up-sampling of the to-be-processed features, and obtaining a segmentation mask; a coordinate point set of the segmentation mask is extracted, and pixel coordinates of the highest point and the lowest point in the y direction are obtained from the coordinate point set; according to the highest point pixel coordinate and the lowest point pixel coordinate in the image A and the highest point pixel coordinate and the lowest point pixel coordinate in the image B, the fuzzy height of the stalk crop is calculated; according to the method, more accurate spatial features and local features are obtained, and the height of the stalk crop is calculated according to the images of two different viewpoints and the actual distance, so that accurate plant height calculation is realized.

Description

Technical Field

[0001] The invention relates to the technical field of composite harvesting of stalk crops, and in particular to a stalk crop height calculation method and a composite harvesting method for stalk crops. Background Art

[0002] The soybean-corn composite harvesting robot is one of the representatives of advanced technologies in the agricultural field. It is specially designed for the harvesting task of composite crops in farmland, making it widely used in modern agriculture. Among them, by adjusting the height of the harvesting platform, different types of crops can be harvested, while reducing the time used for harvesting and improving harvesting efficiency. However, efficient and accurate harvesting operations depend on the control of the height of the harvesting platform. Traditional harvesting control methods mostly rely on human experience, and it is difficult to guarantee the harvesting effect of composite planting. Among them, a single harvesting equipment is suitable for covering a large area of ​​uniformly planted crops, and it is difficult to accurately harvest in the composite planting mode; traditional harvesting equipment uses the method of harvesting soybeans and corn separately, which puts high demands on the planting method and is difficult to apply universally.

[0003] Publication No. CN116883480A is a corn plant height detection method based on binocular image and ground-based radar fusion point cloud. It solves the inaccurate detection result caused by the lack of single-modal point cloud data by fusing the point cloud data reconstructed by binocular image 3D and the point cloud data collected by ground-based radar. It is mainly used to solve the problem of multi-modal fusion of data, and the data processing volume is large and complex. Publication No. CN113554691B is a plant height measurement method. It calculates the plant height by calibrating the camera's internal parameters and retrograde calibration and then combining the acquired real image information. This method adds filtering of point cloud data in the middle to reduce the negative impact of background on data analysis, so as to find the area of ​​interest to calculate the plant height. Since the parameters need to be calibrated, the accuracy is poor and it is labor-intensive and time-consuming. Summary of the invention

[0004] In view of the problems in the prior art, the present invention provides a method for calculating the height of stem crops and a composite harvesting method for stem crops, aiming to improve the segmentation accuracy of images to achieve accurate plant height detection and realize precise harvesting under the composite planting mode.

[0005] The method for calculating the height of stalk crops comprises the following steps:

[0006] Step 1: Shoot at a distance along the same horizontal direction A image and close range B-image;

[0007] Step 2: Perform the same processing on image A and image B respectively through the NF-SAM model. The NF-SAM model includes a ResNet encoder module, a VIT encoder module, a feature fusion module, a Mask decoder module, and a prompting module. Both image A and image B are recorded as images to be processed; the processing process is:

[0008] Step 2.1: Obtain local feature maps of different scales in the image to be processed through the ResNet encoder module;

[0009] Step 2.2: Obtain the global features of the image to be processed through the VIT encoder module and flatten it into a one-dimensional vector;

[0010] Step 2.3: The local feature map and the one-dimensional vector are fully connected and fused through the feature fusion module to obtain the fused feature;

[0011] Step 2.4: The fused features are repeatedly fused and updated with the prompt information obtained from the prompt embedding module through the Mask decoder module to obtain the features to be processed;

[0012] Step 3: After deconvolution and upsampling of the features to be processed, the segmentation mask is obtained;

[0013] Step 4: Extract the coordinate point set of the segmentation mask and obtain the highest and lowest pixel coordinates in the y direction from the coordinate point set;

[0014] Step 5: According to the highest pixel coordinates in image A , lowest pixel coordinates And the highest pixel coordinates in the B image , lowest pixel coordinates , calculate the fuzzy height of the stalk crop : .

[0015] Further, the ResNet encoder module includes four Block blocks and a residual connection module is set between each Block block. The image to be processed is sequentially processed in the first Block block. Convolutional layer, batch normalization layer, ReLU activation layer, Convolutional layer, batch normalization layer, ReLU activation layer, After the convolution layer and batch normalization layer, the local feature map is obtained , whose size is ; The image to be processed jumps to the first residual connection module and the local feature map Perform residual connection and form the first residual feature. The first residual feature obtains the local feature map after passing through the second Block. , whose size is ; The first residual feature jumps to the second residual connection module and the local feature map

[0016] Perform residual connection and form the second residual feature. The second residual feature obtains the local feature map after passing through the third Block. , whose size is ; The second residual feature jumps to the third residual connection module and the local feature map Perform residual connection and form the third residual feature. The third residual feature obtains the local feature map after passing through the fourth block. , whose size is .

[0017] Further: The processing process in the VIT encoder module is:

[0018] Step 2.2.1: The image to be processed Divide evenly into blocks, where S is set to 16, H is the height of the image to be processed, W is the width of the image to be processed, and 3 is the number of channels of the image to be processed; and each block is expanded into a one-bit vector as , size is ;

[0019] Step 2.2.2: Flatten the Blocks are transferred to a The linear projection layer obtains the features .

[0020]

[0021] in, is the linear projection matrix with dimensions ; For the The flattened vector of the blocks, The size of the bias vector is ;

[0022] Step 2.2.3: Add the positional encoding of the same dimension to each block. The positional encoding information :

[0023] in, yes The position encoding information corresponding to the block has a dimension of ;

[0024] Positional encoding It is expressed as:

[0025] in, yes The position encoding information of each patch;

[0026] Step 2.2.4: The feature block containing the position encoding information Input encoder, the encoder contains normalization layer, multi-head self-attention module and multi-layer perceptron; the input feature block is processed by normalization layer Normalize:

[0027] After normalization Input into the multi-head attention module;

[0028] in, , , ; is the learned projection matrix; then a jump connection is made to connect the output of the multi-head attention module with the feature block Add up the features :

[0029] The second layer of the encoder is then performed, which includes normalization and MLP layers. Perform normalization;

[0030] Then the features Send to the multi-layer perception module:

[0031] in: is the weight matrix, is the bias term;

[0032] After the multi-layer perception module is processed, another jump connection is performed , complete the image encoding, after several encodings, obtain the global features of the image and flatten it into a one-dimensional vector .

[0033] Further: The process of full connection fusion is:

[0034] Step 2.3.1: Feature Map Flattened to a one-dimensional vector : ;

[0035] Step 2.3.2: Convert the one-dimensional vector With one-dimensional vector Splice and obtain features :

[0036] The characteristics Fusion is performed through the fully connected layer and features are obtained :

[0037] in, Fully connected layer.

[0038] Further: record the features to be processed as features , then the segmentation mask is:

[0039] in, is the segmentation mask; the coordinate point set of the segmentation mask for:

[0040] The pixel coordinates of the highest point and the lowest point in the direction are: ; .

[0041] The composite harvesting method for stalk crops, based on the stalk crop height calculation method, comprises the following steps:

[0042] Step 1: Get the fuzzy height of the crop stalk ;

[0043] Step 2: Based on the blur height and the parameters in the rule base to obtain the height of the ears of different stalk crops : ;

[0044] The rule base is:

[0045] is the parameter in the rule base;

[0046] Step 3: Based on the height Adjust the cutting height of the harvesting equipment.

[0047] The beneficial effects of the present invention are as follows: the feature extraction parts of the ResNet encoder and the VIT encoder are integrated to more accurately obtain spatial features and local features, and the height of the stem crop is calculated according to images from two different viewpoints and the actual distance to achieve accurate plant height calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a structural block diagram of the method for calculating the height of stalk crops in the present invention; Figure 2 It is a structural diagram of the ResNet encoder module; Figure 3 This is a structural diagram of the VIT encoder module. DETAILED DESCRIPTION

[0049] The present invention is described in detail below in conjunction with the accompanying drawings. The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention. The directional terms such as left, middle, right, top, and bottom in the embodiments of the present invention are only relative concepts or are based on the normal use state of the product, and should not be considered as restrictive.

[0050] The visual camera is located in front of the autonomous harvesting robot as a visual sensor to sense the state of the stalk crops to be harvested in front. The robot first uses the two-dimensional visual camera to visually perceive the stalk crops to be harvested, and calculates the height of the actual stalk crops to be harvested based on the pixel difference of the stalk crops to be harvested at two different angles. Secondly, the optimal harvesting height is calculated in combination with the rule base of the harvesting table.

[0051] Methods for calculating the height of stalk crops, such as Figure 1 As shown, the following steps are included:

[0052] Step 1: In order to fully perceive and record the pixel area of ​​the stalk crops to be harvested at different distances for calculating the actual height of the stalk crops to be harvested, the original driving direction should be maintained when taking the second photo; take photos of the long-distance crop in the same horizontal direction. A image and close range B-image;

[0053] Step 2: Since corn and soybean plants have unique structural characteristics, such as the leaves of corn are arc-shaped and the leaves of soybean are trifoliate leaves, the plants can be better segmented by analyzing the local structural characteristics of these leaves, and the height of the stalk crops can be accurately calculated; therefore, it is necessary to use a dual-stream branch (ResNet encoder module + VIT encoder module) to improve the accuracy of feature capture; that is, the A image and the B image are processed in the same way by the NF-SAM model, which includes the ResNet encoder module, the VIT encoder module, the feature fusion module, the Mask decoder module and the prompt embedding module, and both the A image and the B image are recorded as the images to be processed; the processing process is:

[0054] Step 2.1: Obtain local feature maps of different scales in the image to be processed through the ResNet encoder module;

[0055] like Figure 2 As shown in the figure, the ResNet encoder module includes four Block blocks and a residual connection module is set between each Block block. The image to be processed is sequentially processed in the first Block block. Convolutional layer, batch normalization layer, ReLU activation layer, Convolutional layer, batch normalization layer, ReLU activation layer, After the convolution layer and batch normalization layer, the local feature map is obtained , whose size is ; The image to be processed jumps to the first residual connection module and the local feature map Perform residual connection and form the first residual feature. The first residual feature obtains the local feature map after passing through the second Block. , whose size is ; The first residual feature jumps to the second residual connection module and the local feature map Perform residual connection and form the second residual feature. The second residual feature obtains the local feature map after passing through the third Block. , whose size is ; The second residual feature jumps to the third residual connection module and the local feature map Perform residual connection and form the third residual feature. The third residual feature obtains the local feature map after passing through the fourth block. , whose size is ;

[0056] Step 2.2: Obtain the global features of the image to be processed through the VIT encoder module and flatten it into a one-dimensional vector; Figure 3As shown, the processing process in the VIT encoder module is:

[0057] Step 2.2.1: The image to be processed Divide evenly into blocks, where S is set to 16, H is the height of the image to be processed, W is the width of the image to be processed, and 3 is the number of channels of the image to be processed; and each block is expanded into a one-bit vector as , size is ;

[0058] Step 2.2.2: Flatten the Blocks are transferred to a The linear projection layer obtains the features .

[0059]

[0060] in, is the linear projection matrix with dimensions ; For the The flattened vector of the blocks, The size of the bias vector is ;

[0061] Step 2.2.3: To exploit spatial priors, add positional encodings of the same dimension to each block. The positional encoding information :

[0062] in, yes The position encoding information corresponding to the block has a dimension of ;

[0063] Positional encoding It is expressed as:

[0064] in, yes The position encoding information of each patch;

[0065] Step 2.2.4: The feature block containing the position encoding information Input encoder, the encoder contains normalization layer, multi-head self-attention module and multi-layer perceptron; normalization is to maintain the stability of feature distribution, improve the training efficiency and performance of the model, and reduce the risk of gradient disappearance or explosion in deep network, thereby accelerating the convergence of the model; through the normalization layer, the input feature block Normalize:

[0066] After normalization Input into the multi-head attention module;

[0067] in, , , ; is the learned projection matrix; then a jump connection is made to connect the output of the multi-head attention module with the feature block Add up the features :

[0068] The second layer of the encoder is then performed, which includes normalization and MLP layers. Perform normalization;

[0069] Then the features Send to the multi-layer perception module:

[0070] in: is the weight matrix, is the bias term;

[0071] After the multi-layer perception module is processed, another jump connection is performed , complete the image encoding, after several encodings, obtain the global features of the image and flatten it into a one-dimensional vector ;

[0072] Step 2.3: The local feature map and the one-dimensional vector are fully connected and fused through the feature fusion module to obtain the fused feature;

[0073] The process of full connection fusion is:

[0074] Step 2.3.1: Feature Map Flattened to a one-dimensional vector : ;

[0075] Step 2.3.2: Convert the one-dimensional vector With one-dimensional vector Splice and obtain features :

[0076] The characteristics Fusion is performed through the fully connected layer and features are obtained :

[0077] Among them, FC(.) fully connected layer;

[0078] Step 2.4: The Mask decoder module repeatedly fuses and updates the fused features with the prompt information obtained from the prompt embedding module to obtain the features to be processed. The prompt encoder module allows the operator to prompt the harvesting area in the form of points and boxes to avoid the influence of surrounding plants on the segmentation network and improve the generalization of the network.

[0079] Step 3: After deconvolution and upsampling of the features to be processed, the segmentation mask is obtained;

[0080] Record the features to be processed as features , then the segmentation mask is:

[0081] in, is the segmentation mask; the coordinate point set of the segmentation mask for: ;

[0082] Step 4: Extract the coordinate point set of the segmentation mask and obtain the highest and lowest pixel coordinates in the y direction from the coordinate point set; the highest pixel coordinates and the lowest pixel coordinates in the y direction are: ; ;

[0083] Step 5: According to the highest pixel coordinates in image A , lowest pixel coordinates And the highest pixel coordinates in the B image , lowest pixel coordinates , calculate the fuzzy height of the stalk crop : .

[0084] The composite harvesting method for stalk crops, based on the stalk crop height calculation method, comprises the following steps:

[0085] Step 1: Get the fuzzy height of the crop stalk ;

[0086] Step 2: Based on the blur height and the parameters in the rule base to obtain the height of the ears of different stalk crops : ;

[0087] The rule base is:

[0088] is the parameter in the rule base;

[0089] Step 3: Based on the height Adjust the cutting height of the harvesting equipment.

[0090] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for calculating the height of stalk crops, characterized in that: The following steps are involved: Step 1: Shoot at a distance along the same horizontal direction A image and close range B-image; Step 2: Perform the same processing on image A and image B respectively through the NF-SAM model. The NF-SAM model includes a ResNet encoder module, a VIT encoder module, a feature fusion module, a Mask decoder module, and a prompt embedding module. Both image A and image B are recorded as images to be processed; the processing process is: Step 2.1: Obtain local feature maps of different scales in the image to be processed through the ResNet encoder module; Step 2.2: Obtain the global features of the image to be processed through the VIT encoder module and flatten it into a one-dimensional vector; Step 2.3: The local feature map and the one-dimensional vector are fully connected and fused through the feature fusion module to obtain the fused feature; Step 2.4: The fused features are repeatedly fused and updated with the prompt information obtained from the prompt embedding module through the Mask decoder module to obtain the features to be processed; Step 3: After deconvolution and upsampling of the features to be processed, the segmentation mask is obtained; Step 4: Extract the coordinate point set of the segmentation mask and obtain the highest and lowest pixel coordinates in the y direction from the coordinate point set; Step 5: According to the highest pixel coordinates in image A , lowest point pixel coordinates And the highest pixel coordinates in the B image , lowest point pixel coordinates , calculate the fuzzy height of the stalk crop : 。 2. The method for calculating the height of stalk crops according to claim 1, characterized in that: The ResNet encoder module consists of four Block blocks and a residual connection module is set between each Block block. The image to be processed is sequentially processed in the first Block block. Convolutional layer, batch normalization layer, ReLU activation layer, Convolutional layer, batch normalization layer, ReLU activation layer, After the convolution layer and batch normalization layer, the local feature map is obtained , whose size is ; The image to be processed jumps to the first residual connection module and the local feature map Perform residual connection and form the first residual feature. The first residual feature obtains the local feature map after passing through the second Block. , whose size is ; The first residual feature jumps to the second residual connection module and the local feature map Perform residual connection and form the second residual feature. The second residual feature obtains the local feature map after passing through the third Block. , whose size is ; The second residual feature jumps to the third residual connection module and the local feature map Perform residual connection and form the third residual feature. The third residual feature obtains the local feature map after passing through the fourth block. , whose size is .

3. The method for calculating the height of stalk crops according to claim 2, characterized in that: The processing process in the VIT encoder module is: Step 2.2.1: The image to be processed Divide evenly into blocks, where S is set to 16, H is the height of the image to be processed, W is the width of the image to be processed, and 3 is the number of channels of the image to be processed; and each block is expanded into a one-bit vector as , size is ; Step 2.2.2: Flatten the Blocks are transferred to a The linear projection layer obtains the features , ； in, is the linear projection matrix with dimensions ; For the The flattened vector of the blocks, The size of the bias vector is ; Step 2.2.3: Add the positional encoding of the same dimension to each block. The positional encoding information : ； in, yes The position encoding information corresponding to the block has a dimension of ; Positional encoding It is expressed as: ； in, yes The position encoding information of each patch; Step 2.2.4: The feature block containing the position encoding information Input encoder, the encoder contains normalization layer, multi-head self-attention module and multi-layer perceptron; the input feature block is processed by normalization layer Normalize: ； After normalization Input into the multi-head attention module; ； in, , , ; is the learned projection matrix; then a jump connection is made to connect the output of the multi-head attention module with the feature block Add up the features : ； The second layer of the encoder is then performed, which includes normalization and MLP layers. Perform normalization; ； Then the features Send to the multi-layer perception module: ； in: is the weight matrix, is the bias term; After the multi-layer perception module is processed, another jump connection is performed , complete the image encoding, after several encodings, obtain the global features of the image and flatten it into a one-dimensional vector .

4. The method for calculating the height of stalk crops according to claim 3, characterized in that: The process of full connection fusion is: Step 2.3.1: Feature Map Flattened to a one-dimensional vector : ； Step 2.3.2: Convert the one-dimensional vector With one-dimensional vector Splice and obtain features : ； The characteristics Fusion is performed through the fully connected layer and features are obtained : ； in, Fully connected layer.

5. The method for calculating the height of stalk crops according to claim 1, characterized in that: Record the features to be processed as features , then the segmentation mask is: ； in, is the segmentation mask; the coordinate point set of the segmentation mask for: ； The pixel coordinates of the highest point and the lowest point in the y direction are ; .

6. A composite harvesting method for stalk crops, characterized in that: The method for calculating the height of stalk crops according to claim 1 comprises the following steps: Step 1: Get the fuzzy height of the crop stalk ; Step 2: Based on the blur height and the parameters in the rule base to obtain the height of the ears of different stalk crops : ； The rule base is: ； is the parameter in the rule base; Step 3: Based on the height Adjust the cutting height of the harvesting equipment.

Images